1. Introduction

Haze is a phenomenon in which particles in the air scatter light and obscure an image ^[1]. As a result, outdoor images may have visual limitations. Haze-induced issues can be fatal, especially in situations where traffic conditions require high visibility in real time ^[2], such as car accidents, suspension of aircraft operations, and docking of ships. Various studies have been conducted to remove the haze effects from images to ensure consistent visibility regardless of weather conditions. In addition, during disasters such as fires, it is essential to secure visibility in the event of high smoke levels in the air. Removing the haze effects in images is essential to observe situations through video equipment such as CCTV to cope with disaster situations.

Consistent visible images have many advantages. Enhanced images after haze removal can be used as data for various deep learning applications, such as object recognition and tracking. In deep learning, the learning effect of recognizing objects in images may vary even when the illumination varies at weak points ^[3]. Hence, dehazed images are highly desirable in various fields.

In robot vision, cameras are critical because they are responsible for the visual capabilities. Obtaining clear images from the camera regardless of the weather conditions can determine the performance of robot vision and mobility ^[4]. Dehazing can be used to obtain haze-free real-time images in driving environments and can provide improved visibility during driving and parking ^[5].

Consistently available high visibility can also reduce the probability of accidents in autonomous driving. Additionally, image dehazing can be applied to crime prevention by assisting in identifying the face of a perpetrator in a hazy image. Hence, image dehazing is important in the field of image processing and image enhancement. Because image dehazing is an ill-posed problem, it is necessary to test various approaches based on an atmospheric scattering model to solve it.

Many studies on image dehazing have been performed using convolutional neural networks (CNNs) ^[6-8]. The DehazeNet method ^[6] was used to estimate a transmission map directly from a hazy image. The AOD-Net method ^[7] obtains a dehazed image from a hazy image using CNNs. Recently, various studies have been conducted using methods to estimate feature maps in hazy images to produce haze-free images ^[8].

One method ^[9] uses CNNs to estimate the depth, and another method ^[10] applies transfer learning using a simple encoder-decoder network structure with skip connections. In this paper, we propose a way to estimate the transmission map indirectly from depth estimation using CNNs to generate dehazed images based on an atmospheric scattering model.

2. Related Works

In the past, haze removal was performed using image processing, in which basic contrast enhancement could be used. However, most haze removal methods are based on hypotheses or empirical evidence. For example, the dark channel prior (DCP) method ^[12] is based on the hypothesis that there is a low value in at least one of three RGB channels in color images. Several methods using CNNs that can remove haze within an image were developed with advancements in deep learning techniques. This led to the development of the DehazeNet method ^[6], which is based on an atmospheric scattering model and estimates the transmission map from a hazy image through a CNN. An illustration of the atmospheric scattering model is shown in Fig. 1.

The atmospheric scattering model ^[11] can be represented as:

From Eq. (1), the clean (haze-free) image $J\left(x\right)$ can be expressed as:

In Eq. (1), $I\left(x\right)~ $is the hazy image, $J\left(x\right)$ is the clean image, $t\left(x\right)$is the transmission map, and $\alpha $ is the global atmospheric light. The transmission map $t\left(x\right)$ ^[1,6] can be expressed as:

In Eq. (3), $\beta $ is the scattering coefficient, and $~ d\left(x\right)$ is the depth (distance). Eq. (3) shows that depth information affects the transmission values $t\left(x\right)$.

Typical image dehazing methods estimate the transmission map. In some previous works, the depth information was used for a similar problem of fog removal. Fog effects have been removed using depth estimation, which is based on the assumption that the difference between brightness and saturation becomes larger as the depth becomes larger ^[19]. Depth values have been estimated from the degree of blur for single-image fog removal ^[20]. Unlike previous studies, we propose the application of depth information with deep learning for the estimation of transmission maps. In the proposed method, the depth is estimated using deep learning methods, which give more accurate depth values.

DehazeNet ^[6] is a typical haze removal method that uses CNNs to estimate the transmission map and obtain a dehazed image from it based on an atmospheric scattering model. It requires guided image filtering as a post-processing procedure to refine the transmission map. An advantage of this method is that CNNs are used for deep learning to solve the image dehazing problem.

Unlike the DehazeNet ^[6] method, the AOD-Net ^[7] method combines a transmission map and global atmospheric light parameters into a single parameter function $K\left(x\right)$, which can be learned through deep learning networks. An advantage of AOD-Net is that it is an end-to-end deep learning network. The densely connected pyramid dehazing network (DCPDN) ^[13] is a GAN-based method that can produce an image similar to a dehazed image. In this method, separate networks are used for learning the transmission map and global atmospheric light so that it can generate a haze-free image through a joint discriminator. A recent method called FFA-Net ^[8] learns channel attention and pixel attention maps on a block-by-block basis. It removes the haze by concatenating a hazy image and learns the feature maps as residual networks.

Deep learning networks have been used for depth estimation as well as haze removal. The Monodepth method ^[14] learns from stereo images and can predict depth information from a single image, whereas the Densedepth method ^[10] can estimate depth information using transfer learning. In the Monodepth method ^[14], KITTI data ^[15] are used as stereo images to estimate the disparity map derived from the left image, which is consistent with the right image. The right image requires the disparity map estimated on the left image to calculate the error. As this process repeats, it is possible to create an image in the opposite direction, which allows the creation of stereo images and depth information.

Monodepth2 ^[9] is a follow-up to the Monodepth method ^[14]. It uses the characteristics of the KITTI dataset ^[15], which was constructed using consecutive images captured by a moving car. In Monodepth2, the results can be corrected through reprojection. This method leads to fewer errors because of the creation of stereo images. The Densedepth method ^[10] uses layers consisting of an encoder–decoder structure, which are interconnected using skip connections. In this method, KITTI data ^[15] and NYU Depth V2 data ^[16] can be used to estimate depth information for both indoor and outdoor images.

Fig. 1. Atmospheric scattering model.

Fig. 2. Sequence diagram of the proposed method.

Fig. 3. Network structure of the Monodepth2 method for depth estimation.

3. The Proposed Method

The proposed method generates a transmission map indirectly from a depth map, which can be generated by using previous depth estimation networks based on deep learning. Then, we obtain a dehazed image using a transmission map based on an atmospheric scattering model. Fig. 2 shows a sequence diagram of the proposed method.

As shown in Fig. 2, the depth map is estimated before the estimation of the transmission map. For the depth map, a training process is performed using depth estimation networks based on deep learning. After the training process is complete, a depth estimation model is obtained. Once the depth estimation model is obtained, image dehazing can be performed on a hazy image. For a hazy input image, we estimate the depth map using the depth estimation model. The transmission map is estimated from the depth map using the relationship described in Eq. (3). Finally, we obtain the dehazed image using the atmospheric scattering model described in Eq. (2).

Two methods were tested for the depth estimation model. The first method is Monodepth2 ^[9], which allows the correction of loss values in learning by applying additional information to the network for the depth estimation. The second method is Densedepth ^[10], which uses transfer learning with both indoor and outdoor image data.

The network structure of Monodepth2 method is shown in Fig. 3. The depth network is based on the U-Net structure, which enables the prediction of the overall depth information. The pose network assists in predicting the depth information from the movement of objects that are in the front and rear image frames. Using the information in the pose networks, the networks adjust the parameters to generate a depth map. For this method, training is conducted using the KITTI datasets ^[15], which include mono images, stereo images, and mono and stereo images.

Fig. 4 presents the detailed network structure of the Densedepth method ^[10]. This network was originally applied for image classification, and the encoder–decoder structure method was used to estimate the depth. The training of this method uses the KITTI data and NYU2 depth data. The NYU2 depth data are indoor data, and the KITTI data are outdoor data.

4. Experimental Results

Table 1 presents the parameters used for training Monodepth2 and Densedepth in the experiments.

4.1 Results of Depth Estimation Networks

Fig. 5 shows the experimental results using Monodepth2. Fig. 5(a) shows a test image of the Berkeley dataset ^[17] as the input. Figs. 5(b)-(d) show the resulting depth maps using mono images as the training data. Figs. 5(b) and (c) show the results of training with image sizes of 640 ${\times}$ 192 and 1024 ${\times}$ 320, respectively.

Fig. 5(d) shows the result of training with an image size of 640 ${\times}$ 192, as shown in Fig. 5(b) without applying the pose network. Figs. 5(e)-(g) are the resulting depth map images using stereo images as the training data. Figs. 5(e) and (f) show the results of training with image sizes of 640 ${\times}$ 192 and 1024 ${\times}$ 320, respectively. Fig. 5(g) shows the resulting depth map without applying the pose network with a size of 640 ${\times}$ 192, which is same as the size in Fig. 5(e).

Figs. 5(h)-(j) show the resulting depth maps using both mono and stereo images as the training data. Figs. 5(h) and (i) show the results of training with sizes of 640 ${\times}$ 192 and 1024 ${\times}$ 320, respectively. Fig. 5(j) show the result of training without applying the pose network with a size of 640${\times}$192, which is the same as the size in Fig. 5(h). There is a tendency for small objects to be perceived at farther distances and for large objects to be perceived at nearer distances. If the pose network is not applied, the overall depth outlines are blurred, and the depth estimation values become less accurate.

Fig. 6 shows the experimental results obtained using Densedepth. The encoder part of Densedepth network was set as DenseNet-169 ^[21] for the experiments. We compared the results of depth estimation for indoor and outdoor images using the NYU2 depth dataset and KITTI dataset.

Fig. 6(a) shows the indoor image data ^[16] used as the input for Figs. 6(b) and (c). Fig. 6(d) shows the outdoor image data ^[19] used as the input for Figs. 6(e) and (f). Figs. 6(b) and (e) show the depth map images obtained by training using the NYU2 depth dataset. Figs. 6(c) and (f) show the depth map images obtained by training using the KITTI dataset.

The NYU2 depth dataset provides indoor image data, and the KITTI dataset provides outdoor image data, so the resulting depth maps are different. With the NYU2 depth dataset, the results preserve more edges of objects. The results obtained with the NYU2 depth dataset show more detailed depth results than those obtained with the KITTI dataset for indoor images. For the outdoor images, the results obtained with the NYU2 depth dataset cannot predict the overall depth map, while the results with the KITTI dataset can predict the depth map more evenly.

Fig. 4. Detailed network structure of the Densedepth method (a) Encoder network, (b) Decoder network (AV: average pooling, CC: concatenate, CV: convolution, DB: dense block, GAP: global average pooling, MP: max pooling, SM: softmax, US: up-sampling).

Fig. 5. Results of depth estimation by training using the various configurations of data (a) Input image, (b) Result with mono image (640${\times}$192), (c) Result with mono image (1024 ${\times}$ 320), (d) Result with mono image (640 ${\times}$ 192) without the pose network, (e) Result with stereo images (640 ${\times}$ 192), (f) Result with stereo images (1024 ${\times}$ 320), (g) Result with stereo images (640 ${\times}$ 192) without the pose network, (h) Result with mono and stereo images (640 ${\times}$ 192), (i) Result with mono and stereo images (1024 ${\times}$ 320), (j) Result with mono and stereo images (640 ${\times}$ 192) without the pose network.

Fig. 6. Results of depth estimation by training using the various configurations of the dataset (a) Indoor input image, (b) Result depth map by [10] with NYU2 depth training data, (c) Result depth map by [10] with KITTI training data, (d) Outdoor input image, (e) Result depth map by [10] with NYU2 depth training data, (f) Result depth map by [10] with KITTI training data.

4.2 Transmission Map and Dehazed Image Obtained using the Proposed Method

For depth estimation, we used previously described depth estimation networks ^[9,10]. Both networks were implemented using TensorFlow codes. For the test, unannotated real-world hazy images were used ^[18]. Haze removal experiments were carried out by converting the depth map into the transmission map using the relationship described in Eq. (3). Figs. 7 and 8 show the process of image dehazing using the proposed method.

Figs. 7(a) and 8(a) show the input hazy images ^[18]. Figs. 7(b) and 8(b) show the depth maps from the input images using the depth estimation model with the depth estimation network. Figs. 7(c) and 8(c) show the visualized transmission maps from the depth map. Figs. 7(d) and 8(d) show the dehazed images after the haze removal is carried out from the transmission map using the atmospheric scattering model described in Eq. (2). In these results, the depth value becomes lower for nearby objects, and the transmission value become higher. In addition, objects at farther distances changed more.

Fig. 7. Process of image dehazing using the proposed method (a) Input image, (b) Depth information, (c) Transmission map (visualized), (d) Dehazed image obtained using the proposed method.

Fig. 8. Process of image dehazing using the proposed method (a) Input image, (b) Depth map, (c) Transmission map (visualized), (d) Dehazed image obtained using the proposed method.

4.3 Result Comparison of Proposed Method and DehazeNet

DehazeNet ^[6] directly generates a transmission map from a hazy image using the CNNs. Comparisons of the results for hazy natural outdoor images for the proposed method and the DehazeNet method are shown in Figs. 9-11. Figs. 9(a)-(c) show the hazy images obtained using Bdd100k ^[17]. Figs. 10(a)-(c) show the dehazed images obtained using the DehazeNet method. Figs. 11(a)-(c) show the dehazed images obtained using the proposed method.

In the DehazeNet method, haze effects are sufficiently removed, changes in the intensity levels are high, and the roads are excessively darkened from recognizing the bright parts of the road as haze. In the results obtained using the proposed method, the haze is removed more evenly, and the road parts are dehazed correctly while preserving the intensity levels without any darkening effects.

We also performed experiments using the synthetic objective testing set (SOTS) ^[18] for the comparison of image dehazing results by DehazeNet and the proposed method with Monodepth2 and Densedepth. Figs. 12 and 13 show the results with the PSNR values, which were calculated using the groundtruth images of SOTS. The figures show that the proposed method gives better PSNR results for image dehazing than the DehazeNet method. The dehazed images from DehazeNet are darker than those from the proposed method.

Fig. 9. Hazy Images obtained using the Bdd100k (a) Hazy image of a street, (b) Hazy image of a road with heavy traffic, (c) Hazy image of roadside trees.

Fig. 10. Dehazed images obtained using the DehazeNet method (a) Dehazed image of the street, (b) Dehazed image of a road with heavy traffic, (c) Dehazed image of roadside trees.

Fig. 11. Dehazed Images obtained using the proposed method (a) Dehazed image of the street, (b) Dehazed image of a road with heavy traffic, (c) Dehazed image of roadside trees.

Fig. 12. Image dehazing results with PSNR values obtained by the proposed method and DehazeNet (a) Hazy image of cityscapes, (b) Dehazed image using the proposed method with Monodepth2 (PSNR: 28.30), (c) Dehazed image using the proposed method with Densedepth (PSNR: 29.35), (d) Dehazed image using DehazeNet (PSNR: 27.60), (e) Groundtruth image.

Fig. 13. Image dehazing results with PSNR values obtained by the proposed method and DehazeNet (a) Hazy image of roadways, (b) Dehazed image using the proposed method with Monodepth2 (PSNR: 29.15), (c) Dehazed image using the proposed method with Densedepth (PSNR: 28.50), (d) Dehazed image using DehazeNet (PSNR: 27.99), (e) Groundtruth image.

Fig. 14. Results of a highly-hazed road image with various scattering coefficient values (a) Input image, (b) β = 0.2, (c) β = 0.4, (d) β = 0.6, (e) β = 0.8, (f) β = 1.0, (g) β = 1.2.

Fig. 15. Results of an airport image with various scattering coefficient values (a) Input image, (b) β = 0.2, (c) β = 0.4, (d) β = 0.6, (e) β = 0.8, (f) β = 1.0, (g) β = 1.2.

Fig. 16. Results of a parking lot image with various scattering coefficient values (a) Input image, (b) β = 0.2, (c) β = 0.4, (d) β = 0.6, (e) β = 0.8, (f) β = 1.0, (g) β = 1.2.

Fig. 17. Results of creek between buildings with various scattering coefficient values (a) Input image, (b) β = 0.2, (c) β = 0.4, (d) β = 0.6, (e) β = 0.8, (f) β = 1.0, (g) β = 1.2.

Fig. 18. Results of park image with various scattering coefficient values (a) Input image, (b) β = 0.2, (c) β = 0.4, (d) β = 0.6, (e) β = 0.8, (f) β = 1.0, (g) β = 1.2.

5. Conclusion

In this paper, we proposed a novel technique for image dehazing by indirectly creating a transmission map through the estimation of the depth map as opposed to direct estimation of the transmission map in previous image dehazing methods. The dehazing results using the proposed method were superior to those of previous methods that generate the transmission map directly with post-processing from the input image. However, the proposed method has limitations. First, the dataset covering the road environment in daylight could provide incorrect results for depth map estimation. Second, it is necessary to set the value of atmospheric light adaptively as each test set needs the appropriate atmospheric light value to be estimated for image dehazing. Future research should be directed to resolve these issues.