2.1 Image Processing-based Image Data Augmentation

Data augmentation techniques can be broadly categorized into image processing-based and learning-based approaches ^[6], as illustrated in Fig. 2. Image processing-based data augmentation mainly increases the number of images by transforming characteristics or applying specific filters. For example, color space transformation is robust against color modifications in the same object because of its ability to manipulate color information in various ways. Geometric transformation ^[7] increases the variability of objects through image resizing, rotation, and displacement, which improves the robustness to changes in object form. Finally, the kernel filter ^[8] emphasizes the features of an image by applying specific filters. However, image processing-based data augmentation distorts the information in the image during the transformation process. Moreover, the performance improvement is insufficient for data lacking semantic information about objects obtained from specialized sensors such as synthetic-aperture radar (SAR) and infrared.

Fig. 2. Illustration of different architectures for learning-based image data augmentation.

2.2 Learning-based Image Data Augmentation

Learning-based image data augmentation trains specific domain data to generate or translate into unique style images, such as style transfer and adversarial training. Fig. 2 compares the three architectures commonly used for learning-based image data augmentation. Variational Auto-Encoder(VAE) model ^[9] can generate an image with a distribution similar to the input image using an encoder and a decoder, as illustrated in Fig. 2(a). However, it generates a low-quality image according to the quality of the input image. GAN model ^[10,11] generates images via an adversarial learning process; where the generator generates images of quality that the discriminator cannot distinguish, and the discriminator enhances the ability to distinguish between real and generated images, as illustrated in Fig. 2(b). However, GAN model requires detailed parameter fine-tuning because of the training instability. Finally, as illustrated in Fig. 2(c), Diffusion model ^[12] adds noise to the image and gradually removes the noise during training to obtain data distribution similar to the original, resulting in high-quality images.

3.1 Pixel Space-based Palette

Diffusion models consist of a forward process and a reverse process. The forward process is a Markovian process that iteratively adds Gaussian noise to the original image. In contrast, the reverse process reconstructs noise into the original image, yielding significant flexibility and tractability. Palette is a type of pixel space-based diffusion models that unified the framework for image-to-image translation tasks, colorization, inpainting, uncropping, and JPEG restoration. The Palette trains a reverse process to translate an infrared image from a visible image, which inverts the forward process. Given a noisy infrared image $\overset{˜}{y}$,

the goal is to recover the target infrared image $y_{0}$. Thus, we parameterize network $f_{\theta }\left(x,\,\,\overset{˜}{y},\,\,\gamma \right)$ using the input visible image$~ x$, a noisy infrared image $\overset{˜}{y}$, and the current noise level $\gamma $.

To optimize the loss function, we predict the noise vector $\epsilon $ as follow:

As Palette minimizes the difference at the pixel-level between real images and translated infrared images by using the spatial dimension, it can effectively translate high-quality images with the structures and texture details.

3.2 Latent Space-based BBDM

Although Palette faithfully translates infrared images from visible images retraining detail textures, it has a limitation that generally requires extensive computational resources due to the use of pixel space. To address this issue, latent space-based diffusion models have been developed to train with fewer computational resources. As a latent space-based diffusion model, BBDM can conduct image-to-image translation between two domains using a stochastic Brownian bridge process , providing promising results. The encoder extracts the feature maps of the image and maps to the high-dimensional latent space. Then, diffusion process is progressed based on the schedule of variance in latent space, then the decoder translates into infrared image. The schedule of variance for Brownian bridge $\delta _{t}$ diffusion process can be designed as

where $T$ is the total steps of the diffusion process. The sampling diversity can be tuned by the maximum variance $\delta _{max}$ at the middle step $t=~ \frac{T}{2}.$ To translate infrared images from visible images, object function of BBDM is as follows:

where $x_{0}$ and $y$ denote initial visible image status and infrared image, and $\epsilon _{\theta }$ is the trained model to estimate ${\epsilon}$. BBDM effectively represents high-dimensional characteristics in latent space via encoder rather than pixel space with the diffusion process, thereby improving the learning efficiency and model generalization.

4.1 Experimental Settings

We evaluate infrared object detection performance using the FLIR dataset ^[15], which contains pairs of well-aligned visible and infrared images from real cameras concluding over 375,000 annotations. The visible images were translated into infrared images, and then the dataset was constructed in various ratios with translated images and FLIR dataset images based on the diffusion model. We use Yolov5l-TA model ^[16] for object detection, which exhibits superior detection performance and is actively used in many industry fields because of its outstanding inference speed. The qualitative metric for the object detection performance calculates the mean Average Precision ($\mathrm{mAP}$) across different Intersection over Union (IoU) thresholds. Furthermore, the object detection model in different settings was trained to analyze the effectiveness of the number of images. Table 2 compares the object detection performance. Besides, we categorized the size of the object based on the pixel scale range of width $W$ and height $H$. Then, the $\mathrm{mAP}$ metrics were classified into $\mathrm{mAP}_{s},$ $\mathrm{mAP}_{m}$, and $\mathrm{mAP}_{l}$ as small, medium, and large, as shown in Table 3. Training with 2,000 and 3,000 images yielded equal results, with a performance of 62.6%. This indicates that the object detection model achieves saturation in terms of performance; the diversity of the FLIR dataset is sufficient even with 3,000 images. Therefore, we set the baseline for training the object detection model to 3,000 in all experiments.

Table 3. The categories of object size within the pixel scale range.

4.2 Quantitative & Qualitative Evaluation

We evaluated quantitative image-to-image translation performance based on the diffusion model using four metrics: Peak Signal-to-noise ratio (PSNR), Structural Similarity (SSIM), inception score (IS) ^[17], and Fréchet Inception Distance (FID) ^[18]. Higher PSNR, SSIM, and IS scores indicate better performance, whereas a lower FID score implies better performance. Table 4 quantitatively compares performance using pixel space-based Palette and latent space-based BBDM. BBDM yields higher PSNR and SSIM scores than Palette, indicating that the translated infrared images have similar structures and characteristics with infrared ground truths. In contrast, Palette achieves a higher IS score than BBDM, implying that high quality and superior diversity of infrared images are translated effectively. Furthermore, The Palette yields a significantly higher FID score than the BBDM. This indicates that Palette preserves detailed information and effectively translates infrared images from visible images.

Fig. 4 shows the qualitative comparison of the translated infrared image from the visible image based on the diffusion models. Palette showed outstanding results in object detection with multiple objects and particular texture retention in infrared images. Additionally, Palette captures spatial information in pixel space and preserves detailed texture for high-quality infrared images. In contrast, BBDM generates poor visual artifacts that result in a failure to detect objects such as trees and cars. This is because the model focuses only on the semantic information, which is the texture information lost during diffusion process. These results indicate that BBDM achieves higher PSNR and SSIM scores; however, it changes the appearance of the object and eventually degrades the object detection performance. In contrast, Palette translates the distribution of the translated infrared images to that of the real infrared image distribution, resulting in promising results in terms of IS and FID scores.

Fig. 4. Qualitative comparison of the translated infrared image.

4.3 Object Detection Performance Based on Mixed Ratio $\mathbf{\lambda}$

Table 5 compares the quantitative object detection performance of the training datasets with various mixed ratios $\lambda $ of the translated infrared images. We also marked the relative value increased from the baseline score in parentheses for better comparison. Furthermore, mixed ratio ${\lambda}$ of 100% refer to the entire translated infrared images using diffusion models. In mixed ratio ${\lambda}$ of 20%, Palette resulted 62.8% $\mathrm{mAP}_{0.5}$, which relatively improved by 0.3% based on the baseline $\mathrm{mAP}_{0.5}$. This indicates that the increase of $\mathrm{mAP}_{0.5}$ improves the object detection performance with superior generalization ability. However, increment of the mixed ratio $\lambda $ degraded $\mathrm{mAP}_{0.5}$ due to the failure of full coverage with the real infrared images distribution, e.g., mixed ratio ${\lambda}$ of 100%. The BBDM exhibits tendencies similar to Palette. BBDM with a mixing ratio ${\lambda}$ of 10% achieved 62.9% $\mathrm{mAP}_{0.5},$ demonstrating a relative improvement of 0.5%. Specifically, the significant decrease in object detection of BBDM to Palette is the hazy appearance yielded by undesirable discrepancies in the distribution of real infrared images.

4.4 Instance-level Analysis

We analyze the effectiveness of instance-level object detection performance by training datasets with various mixed ratios ${\lambda}$ of the translated infrared images using Palette and BBDM. Tables 6 and 7 compare the object detection performance of object size- and class-level, respectively. With the mixed ratio $\lambda $ set to 20%, 30%, and 50%, Palette improves the $\mathrm{mAP}_{s}$, $\mathrm{mAP}_{m}$, and $\mathrm{mAP}_{l}$ compared to the baseline because it considers the statistical characteristics of the infrared images. Furthermore, $\mathrm{mAP}_{car}$ consistently increases as the mixed ratio ${\lambda}$ increases in Table 7. This indicates that the structures and texture details of the car are well captured by Palette due to

the relatively simple shape. In addition, $\mathrm{mAP}_{0.5}$ had no significant impact, for the large proportion consist of small classes of person and car in the experimental dataset, as shown in Fig. 5. In contrast, the object detection performance of BBDM degraded as the mixed ratio $\lambda $ increased. BBDM failed to preserve representation ability during the diffusion process of translating infrared images, thereby demoting the infrared image quality and object detection overall performances.

Fig. 5. Ratio by object size and class.

4.5 Distribution Analysis

We verify the effectiveness of the relationship between the distributions of the real infrared images and translated infrared images from visible images. In particular, we utilize the Uniform Manifold Approximation and Projection (UMAP) ^[19] dimension reduction technique for the visualization of high-dimensional feature of image in low dimensional space. Fig. 6 illustrates the results of the real and translated infrared images distribution. Palette overlaps the distribution of real infrared images than BBDM. This overlap result shows the distribution of the translated images is similar to the real infrared images, which indicated that Palette results have better quantitative and qualitative performances.

Fig. 6. The results of translated infrared images and Ground truth distribution.