1. Introduction

Text-to-image models are designed to generate images that correspond to the content of text descriptions used as inputs. Following the advancements in diffusion models, the development of text-to-image models has accelerated ^{[1- 3]}. Imagen ^[1] utilizes a large-scale transformer-based language model to comprehend text and leverages the strengths of a diffusion model to produce high-fidelity images. Stable Diffusion ^[2] trains a diffusion model in a learned latent space, allowing for efficient image generation from this space in a single network pass. DALLE-3 ^[3], based on the diffusion-driven DALLE-2 ^[4], improves inaccurate captions in the training dataset to generate images that better match the prompts. The development of such text-to-image models has made it easier and more convenient to create images that align with user intentions and are of high quality.

Moreover, as the capability to generate high-quality images has improved, research has not only focused on the creation of images but also on utilizing these generated images in deep learning models to further enhance their applications. DIRE (Diffusion Reconstruction Error) ^[12] measures the error between input images and their reconstructed counterparts generated by diffusion models. LGIQA ^[14] was developed to assess the quality of generated images. GenImage ^[13] is a benchmark dataset created using diffusion models and generative adversarial networks to synthesize images for the detection of generated images. However, previous studies have primarily concentrated on comparing generated images with existing images, with minimal efforts directed toward utilizing these generated images as an alternative to image classification datasets.

Therefore, this study has generated an image dataset using text-to-image models and assessed the accuracy of image classification. The generation of images via text-to-image models is significantly influenced by the composition of the prompts. Unlike prior research ^[19] that directly crafted text and qualitatively evaluated object detection, this study employed texts from Large Language Model (LLM) like ChatGPT4 ^[17] and COCO2017 ^[10] captions for constructing texts and utilized classification for evaluation, significantly enhancing the objectivity and reliability of results. Furthermore, while constructing a generated image dataset for ten animal categories adopted from the COCO2017 dataset, this study evaluated the appropriateness for the classification task using the CLIP Score ^[15] to refine the dataset. Additionally, to prevent issues such as resolution degradation, a deblurred dataset was also developed and tested, where deblurring processing was applied to the generated images. This research contributes by generating datasets from various aspects: the type of generative models, prompt composition methods, and refining strategy through both qualitative and quantitative measures. Furthermore, this work is significant as it employs a universal and objective approach by conducting classification without the consideration of bounding boxes, eliminating subjective interpretations of accuracy.

The main contributions of this study are summarized as follows:

2. Related Work

3. Data

In this study, the two-step process was conducted using the text-to-image model as shown in Fig. 1: first, generating a dataset (Subsection 3.1), and then evaluating it (Subsection 3.2). Additionally, experiments on the image classification task are detailed, dividing datasets into base and deblurred categories. The potential is also explored for extension to segmentation tasks. A preview of the generated dataset can be seen in Fig. 2.

3.1. Data Generation

3.1.1 Environment

This chapter introduces the models used for constructing and preprocessing the empirical data. Among the representative text-to-image models, this study utilized the diffusion-based models Stable Diffusion ^[2] and DALLE-3 ^[3]. To generate the dataset, this paper used Stable Diffusion on a T4 GPU and DALLE-3 within the Bing Image Creator, a free tool provided by Microsoft. Both models were configured to generate images using their zero-shot capabilities. Restormer ^[16] for deblurring was implemented in a T4 GPU environment.

3.1.2 Dataset for image classification

The generated images depend on the type of model used and the composition of the prompt. Even with the same model and prompt, different images can be generated each time. Considering these conditions, this paper constructed a dataset by generating multiple images using two types of models and prompt configurations. Firstly, for the generative models, this study used Stable Diffusion ^[2] and DALLE-3 ^[3], both diffusion-based models, but with different structures. Stable Diffusion utilizes an autoencoder, while DALLE-3 directly predicts pixel values using a diffusion model. Notably, DALLE-3 excels in the prompt following, thus the originality of the prompt significantly influences the images it generates. This difference becomes more pronounced when compared to Stable Diffusion, which is particularly strong with realistic prompts.

Fig. 1. Overview of generating and filtering process.

Fig. 2. Dataset preview.

Secondly, to leverage the differences mentioned above, the prompts are structured using two methods: one utilizing an LLM ^[17] and the other using captions from the COCO2017 ^[10] dataset.

To construct a dataset for classification, it is necessary first to define the classes and incorporate them into the prompt text. Experiments are conducted on ten animal categories from the COCO2017 dataset for classification: cat, dog, horse, bird, zebra, sheep, cow, elephant, bear, and giraffe. For constructing prompts with LLM, these ten animal classes are defined as $C = \{c_1, c_2, c_3, \dots, c_{10}\}$. ChatGPT4 ^[17] was used as the LLM to insert prompts, and a sentence is employed like "Create ten text prompts for a text-to-image model. The object must be $C$, depicting $C$ performing human actions realistically." to generate prompts with greater freedom. While generating original prompts through an LLM, captions from COCO2017 were used to construct reliable prompts that have been created and verified by humans. To align with the objectives of the classification task, it is essential to refine captions such that they specify only one type of animal category and do not feature overlapping animal categories. Thus, for the experiments, only those captions from COCO2017 that contained a distinct animal category $C$ and had no overlapping categories were refined and used. Sentences generated using the LLM or refined from captions were then used as inputs for Stable Diffusion and DALLE-3 to generate more than four images per prompt.

In CIFAR-100 ^[7], a representative dataset for image classification, it is stated that the class an image belongs to should rank at the top of the possible answers to the question "What is in this picture?" Accordingly, datasets crafted to these standards are refined. Firstly, the similarity between images and class labels is checked using the CLIP Score to ensure images are categorized into the intended classes. For the previously defined ten animal classes $C$, the class label is defined as "a photo of a $C$". An image is considered well-generated if the class ranked highest by the CLIP Score ^[15] corresponds to the intended class of the image. Secondly, images generated contrary to the intended design were evaluated qualitatively. Images that were generated in black or mixed with humans and animals were also refined. The process of dataset generation and filtering can be observed in Fig. 1.

3.1.3 Deblurring dataset for image classification

Stable Diffusion ^[2] and DALLE-3 ^[3] are diffusion-based models, characterized by a process that regenerates images from noise, which may necessitate consideration of resolution issues. There are various methods to address resolution issues, among which deblurring, commonly used to remove blurring that frequently occurs in real-world settings, was adopted.

Therefore, this study applied deblurring to existing generated datasets to assess its impact when performing classification tasks. Using datasets refined through quantitative and qualitative evaluations, deblurring was additionally carried out using the Restormer ^[16] model.

4. Results

To determine whether the generated and refined datasets could be practically used in image classification, image classification tests were performed separately on the original generated base dataset and the generated deblurred dataset with added deblurring. The experiments were conducted considering the type of generative model and the composition of the prompts used to create the images, and the results were tabulated. Images for ten different animal categories were generated using various models and prompt types, and the dataset refined through quantitative and qualitative evaluations can be briefly viewed in Fig. 2. The results of experiments conducted using this dataset are presented in Tables 1-4.

Table 1 presents the results of image classification performed on the generated dataset created using prompts from a large language model (LLM). The classification accuracy for images created using Stable Diffusion ^[2] and DALLE-3 ^[3] across ten animal categories averaged 99.18% and 87.25%, respectively, confirming that Stable Diffusion had significantly higher classification accuracy. This trend is also evident in Table 3, which presents the results under deblurring conditions. These results suggest that images generated by DALLE-3, which better followed the unique prompts generated by LLM, were relatively more challenging to classify. Table 2 shows the image classification results for the generated dataset created using captions as prompts. With an average classification accuracy of 99.22% across ten animal categories, the performance is comparably high and does not significantly differ from the 99.18% achieved in Table 1 using LLM text with the same Stable Diffusion model.

Tables 3 and 4 present the results of evaluations performed with added deblurring, considering the resolution due to the nature of the two generative models that regenerate images from noise. When compared with data not subjected to deblurring under the same conditions, the results from Stable Diffusion showed a 0.47% increase in accuracy based on LLM text and a 0.08% increase in accuracy based on caption text. Additionally, results from DALLE-3 showed a decrease of 0.5% in accuracy based on LLM text. This study found that deblurring had a modest effect on improving classification performance for images generated by Stable Diffusion, which excels at creating realistic images. However, it had no significant effect on images generated by DALLE-3, known for its strong prompt-following and originality. This indicates that deblurring techniques are more suited for realistic images and may reduce performance when applied to generated datasets that strongly follow prompts.

Failure cases in image classification tasks can be seen in Fig. 3. A failure case occurs when the class evaluated qualitatively and the class assessed by the YOLOv8 model differ, meaning the image was misclassified. Upon examining the example images, they visually appear to be classified as cat, dog, sheep, and zebra, consistent with the ground truth. However, the predictions incorrectly classify them as dog, cat, dog, and None, indicating misclassifications or the absence of the corresponding object classes.

Fig. 3. Failure cases of classification.

Furthermore, to assess scalability in other tasks, an image segmentation test was also conducted using the YOLOv8-seg model. Partial results of the segmentation can be visually examined in Fig. 4. Upon reviewing the example images, considering the variety of animal categories, the generative models used, and the diversity of prompts, they consistently exhibit high performance, demonstrating potential for use as an actual dataset.

Fig. 4. Visual results of segmentation task.

5. Conclusion

This study presents a novel framework to construct an image classification dataset by using images generated from text-to-image models, employing textual inputs from LLM prompts and COCO2017 captions. Unlike traditional datasets, this research leverages generative models to produce labeled image data specifically tailored for classification tasks, offering a cost-effective and scalable solution. The proposed framework refines the dataset generated by ranking images based on class relevance using quantitative metric such as CLIP Score. This approach significantly enhances data quality while reducing uncertainty. Evaluation results indicate that DALLE-3, which faithfully generates images according to unique LLM prompts, relatively complicates image classification. Moreover, the study notes that the impact of deblurring generated images is generally negative and that a specialized approach is necessary to improve the resolution of generated images. This research suggests that the construction of the generated dataset can be universally applied and potentially expanded from classification tasks to segmentation tasks in future studies. By demonstrating the utility and limitations of using generative models for dataset generation, this study contributes to advancing methodologies in image classification.