2.1 Progressive Image Dehazing

The image dehazing problem involves obtaining a clean image estimate $\hat{J}\left(x\right)$ from an input hazy image $I\left(x\right)$. If we have a depth estimate $\hat{d}\left(x\right)$ and a global atmospheric light estimate $\hat{A}$, we can roughly calculate $\hat{J}\left(x\right)$ for image dehazing with an arbitrary value for $\hat{\beta }$, as shown in Eq. (3), which can be deduced from Eqs. (1) and (2).

To compute $\hat{J}\left(x\right)$ from the depth estimate $\hat{d}\left(x\right)$ accurately, we must accurately obtain the attenuation coefficient estimate $\hat{\beta }$, which determines the degree of haziness. We do not have prior knowledge of the degree of haziness $\hat{\beta }$ for an outdoor hazy image. We propose estimating the $\hat{\beta }$ value progressively using a small attenuation value $\beta _{s}$ for one iteration. If the dehazing operation with $\beta _{s}$ in Eq. (3) is performed for a hazy image $I\left(x\right)$, a slightly dehazed image is obtained. If the dehazing operation is performed again for the dehazed image with $\beta _{s}$, it is equivalent to performing the dehazing operation for the original hazy image $I\left(x\right)$ with 2$\beta _{s}$. To generalize this concept with multiple dehazing iterations, we consider the dehazed image for the $i$th iteration with $\beta _{s}$.

$\hat{J}_{i}\left(x\right)$ represents an estimate of the dehazed image for the $i$th iteration of progressive image dehazing:

In Eq. (4), $\beta _{i}$ represents the cumulative attenuation coefficient for the $i$th iteration in progressive image dehazing:

Eq. (4) means that the dehazed image $\hat{J}_{i}\left(x\right)$ can be effectively obtained from the input hazy image $I\left(x\right)$ using the depth estimate $\hat{d}_{i-1}\left(x\right)$ with attenuation coefficient $i\beta _{s}$. Because we do not have prior knowledge of the attenuation coefficient, $\hat{J}_{i}\left(x\right)$ must be computed progressively from the previous dehazed image $\hat{J}_{i-1}\left(x\right)$, not directly from $I\left(x\right)$. For progressive image dehazing, we define the computation of $\hat{J}_{i+1}\left(x\right)$ from $\hat{J}_{i}\left(x\right)$ with attenuation coefficient $\beta _{s}$ as follows:

where

$\hat{d}_{i,c}\left(x\right)$ in Eq. (7) represents the compensated depth map in the $i$th iteration for progressive image dehazing for $\hat{J}_{i+1}\left(x\right)$ from $\hat{J}_{i}\left(x\right)$ with attenuation coefficient $\beta _{s}$.

The initial value of $\hat{J}_{i}\left(x\right)$ is set as $\hat{J}_{0}\left(x\right)$ = $I\left(x\right)$. From Eqs. (4) and (6), we have:

From Eqs. (5), (7) and (8), we derive:

From Eq. (5),

From Eqs. (9) and (10), we obtain:

The DE for $\hat{d}_{i}\left(x\right)$ is performed using the progressively dehazed image for $\hat{J}_{i}\left(x\right)$ for the next iteration. Because the hazy image is dehazed progressively, the accuracy of the depth estimate can be improved incrementally up to the optimal state. Hence, the image dehazing performance can be effectively improved using the updated depth estimate $\hat{d}_{i}\left(x\right)$ as in Eq. (11) rather than using the initial depth map $\hat{d}_{0}\left(x\right)$ from hazy image $I\left(x\right)$. By using the compensated depth map $\hat{d}_{i,c}\left(x\right)$ in Eq. (7) for the progressive image dehazing operation in Eq. (6), the dehazed image $\hat{J}_{i+1}\left(x\right)$ can be effectively obtained from the input image $I\left(x\right)$ using the depth estimate $\hat{d}_{i}\left(x\right)$ with the attenuation coefficient $\left(i+1\right)\beta _{s}$ in Eq. (11), as in the previous iteration in Eq. (4).

2.2 Depth Estimation Module

The depth estimation (DE) is performed using a deep learning network. One of the following deep learning networks was adopted and tested for the proposed PDDE method:

· Monodepth2 ^[8] using self-supervised monocular training

· DenseDepth ^[9] using pre-trained DenseNet ^[17] for the encoder

· DPT-Hybrid ^[11] using pre-trained ViT ^[18] with the ImageNet-1k dataset ^[19]

2.3 Atmospheric Light Estimation Module

The global atmospheric light $A$ in Eq. (1) is assumed to be homogeneous and constant in the image. We use the U-Net ^[20] structure with an encoder and decoder as DCPDN ^[21]. The encoder is composed of four contracting blocks with convolution (Conv), batch normalization (BN), and rectified linear unit (ReLU) layers. The decoder is composed of four expanding blocks with deconvolution (DConv), BN, and ReLU layers. After the encoder and decoder operations are performed, the atmospheric light estimate $\hat{A}$ is obtained via convolution, pooling, and linear layer operations.

2.4 Optimality Assessment

After progressive image dehazing is performed in each iteration, the dehazed image needs to be assessed to determine whether it has reached the optimal state of dehazing. For optimality assessment, we adopt the entropy ^[22] as a non-reference image quality metric as a measure of information entropy ^[23]. We argue that entropy can be effectively used to measure the relative quality of dehazed images from a single hazy image.

The entropy $H^{C}$ for each channel $C$ (R, G, B) with eight bits (256 levels) can be defined as:

where $p_{j}^{C}$ represents the probability of the intensity level $j$ of channel $C$. The entropy $H$ for an image with R, G, and B channels is defined as the average value of $H^{R}$, $H^{G}$, and $H^{B}$.

Algorithm 1. PDDE algorithm.

Algorithm 1 presents the proposed PDDE algorithm with optimality assessment, including detailed pseudocode for the overall architecture shown in Fig. 1. Let $H_{i}$ represent the entropy value of dehazed image $\hat{J}_{i}\left(x\right)$ for optimality assessment in the $i$th iteration. Because image dehazing is performed at each iteration with $\beta _{s}$, the entropy $H_{i}$ would be increased to the optimal state. However, the entropy $H_{i}$would be decreased if the dehazing operation is performed excessively (over-dehazed). We observed that the entropy $H_{i}$ is approximately a convex function of $i$, which is maximized for the optimally dehazed image in most experimental cases. Hence, we set the optimal state for the iteration where the entropy $H_{i}$ is the maximum value in a predefined index range of 0 to $S_{\max }$. After the proposed algorithm finds the optimal state from the entropy, it obtains the optimal dehazed image and depth map in the optimal state.

3.1 Progressive Image Dehazing

The image dehazing performance of the proposed PDDE method was compared with that of state-of-the-art (SOTA) methods. The DPT-Hybrid method ^[11] was fine-tuned with the RESIDE-v0 outdoor training set (OTS) ^[6] and used for DE in the proposed PDDE method. The training data were composed of 8,970 non-hazy data in OTS and 313,950 synthesized data with atmospheric light $A$ ${\in}$ {0.8, 0.85, 0.9, 0.95, 1.0} and $\beta $ ${\in}$ {0.04, 0.06, 0.08, 0.1, 0.12, 0.16, 0.2}. The input images were resized to 256${\times}$256 instead of being cropped.

Fig. 2 shows the image dehazing results for RESIDE-v0 outdoor images. The proposed PDDE method provided a clearer image than that obtained by the SOTA methods while maintaining color information. For the image in row 1, the proposed method provided results similar to the ground truth (GT) image. For the image in row 2, the GT image appeared relatively hazy, and the proposed PDDE method provided a clearer result. Figs. 3 and 4 show the dehazed images at different iterations of the proposed PDDE method for row 1 and row 2 in Fig. 2(f), respectively. The input hazy images were synthesized from the GT images with $\beta $ = 0.12.

Fig. 5 shows the entropy, peak signal-to-noise ratio (PSNR), and Structural Similarity (SSIM) index values for the iterations of the PDDE method. The optimal state was set for the iteration when the maximum entropy occurred: iteration 25 in Fig. 5(a) and iteration 38 in Fig. 5(b). The images at 10 and 20 iterations were in under-dehazed state (still hazy), whereas those at 40 and 50 iterations were in over-dehazed state, as shown in Figs. 3 and 4.

In row 1 of Fig. 2(g), the GT image is clear. In this case, the entropy, PSNR, and SSIM values were maximized at iteration 25, as shown in Fig. 5(a). In row 2 of Fig. 2(g), the GT image is relatively hazy. In this case, the entropy was maximized at iteration 38, whereas the PSNR and SSIM values were maximized at iteration 26. This means that the dehazed image in iteration 26 is closer than the dehazed image at iteration 28. However, the dehazed image at iteration 38 in Fig. 4(e) is clearer than the dehazed image at iteration at iteration 26 in Fig. 4(d). Notably, the GT image in row 2 of Fig. 2(g) looks relatively hazy in this case of the dataset. The proposed PDDE method provided a dehazed result that was clearer than the hazy GT image in this exceptional case. The dehazed image results in the case of Fig. 4 imply that the entropy can be used as a non-referenced image quality assessment metric.

Fig. 2. Image dehazing results for RESIDE-v0 outdoor images: (a) input hazy image; (b) DCP[1]; (c) AOD-Net[5]; (d) FFA-Net[24]; (e) MSBDN-DFF[25]; (f)PDDE (proposed); (g) GT image.

Fig. 3. Dehazed images at different iterations of the PDDE method inFig. 2(f)(row 1): (a) input hazy image ($\boldsymbol{\beta }=0.12$); (b) 10 iterations; (c) 20 iterations; (d) 25 iterations; (e) 40 iterations; (f) 50 iterations.

Fig. 4. Dehazed images at different iterations of the PDDE method inFig. 2(f)(row): (a) input hazy image ($\beta =0.12$); (b) 10 iterations; (c) 20 iterations; (d) 26 iterations; (e) 38 iterations; (f) 50 iterations.

Fig. 5. Entropy, PSNR and SSIM values at different iterations of the PDDE method: (a) the case of Fig. 3; (b) the case of Fig. 4. Horizontal axis: iteration index. Vertical axis: PSNR (blue, left), SSIM (green, left), entropy (red, right).

Fig. 6 presents a comparison of the image dehazing results for an RESIDE-RTTS outdoor image ^[6], which did not have a GT image. As shown, the proposed PDDE method provided clearer and subjectively better results than the SOTA methods.

For a quantitative comparison, 500 outdoor images from the synthetic objective testing set (SOTS) ^[6] were used for testing. Table 1 presents a quantitative comparison of the proposed PDDE method and the SOTA methods for the SOTS. As shown, the proposed method provided the best results for the PSNR and entropy.

Table 2 presents the performance evaluation of ALEM for the estimation of atmospheric light $\hat{A}$ for different values of $A$ by the mean absolute error (MAE). The ALEM estimates atmospheric light, which determines the luminance and color level of atmosphere. In this experiment, hazy images with 5 values of $A$ were generated from each clean image of SOTS. Equal numbers of hazy images for each value of $A$$\in$ {0.8,0.85,0.9,0.95,1.0} were generated for training. Table 2 shows that the MAE values between $A$ and $\hat{A}$ are close. This means that the ALEM performance is consistent with different values of $A$.

Fig. 6. Image dehazing results for an RESIDE-RTTS outdoor image: (a) input hazy image; (b) DCP; (c) AOD-Net[5]; (d) FFA-Net[24]; (e) MSBDN-DFF[25]; (f) PDDE (proposed).

3.2 Depth Estimation

For a comparison of the depth estimation (DE) from a single hazy image, hazy images were synthesized by Eq. (1) using 652 images, which are data split of Eigen $et~ al$. ^[10] from the KITTI dataset ^[15] with depth information. For the synthesis of 1,956 hazy images, the atmospheric light was estimated using the image processing method by Peng $et~ al$. ^[13], and the attenuation coefficient values of $\beta $${\in}${0.02, 0.04, 0.06} were applied.

The attenuation coefficient $\beta $values were relatively small because the KITTI dataset had depth values up to 80 m, whereas the RESIDE-v0 outdoor dataset had scaled depth values of 0 to 5 (without unit). As the depth range increased, the transmission was decreased by Eq. (2), and the degree of haziness was increased for the same $\beta $ value.

Fig. 7 presents the DE results for hazy images with $\beta $ values of 0.02, 0.04, and 0.06. As shown, the proposed PDDE method yielded the best DE result among the methods examined, especially for the objects with large depth (distance) in the red box, which are clearly visible in the inverted depth maps in each row 2 of Fig. 7(c). These objects were not visible in the hazy image or the depth map of Fig. 7(a). The DE and image dehazing results of the PDDE method were better than those shown in Fig. 7(b), with separate application of image dehazing (FFA-Net) and DE (Monodepth2).

Table 3 presents a comparison of the DE results of the proposed PDDE method and the original DE modules for hazy images. We compared the standard error metrics for DE (a smaller value is better) ^[8,10]. As shown, the proposed method achieved a significantly higher accuracy of DE for hazy images with a large improvement ratio (IR) than the previous DE methods.