2.1 Patch Embedding

Before the emergence of vision transformers ^[1] by Dosovitsky et al., CNN models ^[6,19] were used widely in computer vision. Subsequently, the emergence of vision transformer DL models based solely on self-attention operations has become dominant, excluding the convolution structure ^[1,3,5,9]. The concept of self-attention in computer vision is similar to self-attention in the field of NLP; however, it is possible to change the concept of image-patch in vision transformer models to that of sentence-word in NLP through the patch embedding process ^[1], as shown in Fig. 1. This concept is simple. It cuts the image into a non-overlapping 16${\times}$16 patch for linear projection and adds a class token. Through these ideas, vision transformers can achieve state-of-the-art performance not only in NLP but also in computer vision.

On the other hand, despite the contributions of patch embedding, most studies on computer vision tasks have focused on improving model architectures and computational methods, such as convolution, multi-layer perception, and self-attention ^[5-8, ^30]. Accordingly, these studies cannot overcome the limitations of using only the spatial information of the image. As shown in Fig. 1, when patch embedding is performed, the patches cut into 16${\times}$16 pixels are converted to a 196${\times}$C-dimensional matrix through a 2D-convolution operation. Owing to the nature of the CNN, the weights of the filters applied to each patch are shared. Therefore, it may be helpful to match the uniform format for each patch rather than to use the original image of the pixel.

Fig. 1. Diagram showing the patch embedding process of a sample image of size 224${\times}$224 extracted from imageNet-1k dataset. The RGB image is cut into 196(=14${\times}$14) 16${\times}$16 size patches, and is released in the form of a matrix of 196${\times}$C size through a non-overlapping 2D convolution operation of the 16${\times}$16 size filter.}

2.2 2D-Discrete Cosine Transform

The 2D-DCT is used widely in signal processing, and various fields, including image compression, such as JPEGs ^[20]. One of the advantages of 2D-DCT is that the image can be viewed from a frequency perspective. When the 2D-DCT (in units of N${\times}$N block size) is performed on the image, the upper-left side of the block has low-frequency information, whereas the lower-right side has high-frequency information. Fig. 2 presents the result from obtaining an image with frequency information for each block by performing the N${\times}$N block 2D-DCT on the RGB image with a resolution of 224${\times}$224 in Fig. 1.

An image is a signal in which spatial redundant information is captured. Therefore, if the 2D-DCT is performed on a block basis, the frequency and spatial information can be expressed in the local and global parts, respectively. The motivation of this study is that by exploiting these advantages, the vision transformer models without inductive bias can be better trained by inputting images with the 2D-DCT applied to vision transformer models. The 2D-DCT with the N${\times}$N size can be calculated as follows:

where D represents one N${\times}$N-sized block. For example, the original image of a 224${\times}$224 resolution may be converted to 3136 4${\times}$4 size blocks, 784 8${\times}$8 size blocks, 196 16${\times}$16 size blocks, or one 224${\times}$224 block. In (1), i and j are the pixel indices of the blocks converted by the 2D-DCT, and x and y are the pixel indices of the original block I. $\alpha $ is a scale factor for ensuring that the transform is orthonormal and is given as follows:

In this way, the original image can be converted to a frequency domain on a block basis, and within the block, the upper-left part can concentrate low-frequency energy. The lower right can have the relatively less important high-frequency energy.

Fig. 2. Results of a 2D-discrete cosine transform (DCT) operation for the 224${\times}$224 size input image of Fig. 1 in block units of (a) 4${\times}$4; (b) 8${\times}$8; (c) 16${\times}$16; (d) 224${\times}$224.}

2.3 Vision Transformer

Fig. 3(a) shows the overall structure of the vision transformer used for image classification. First, the input image resized to a 224${\times}$224 resolution was cut into 196 16${\times}$16 size patches through a patch embedding process. Subsequently, each patch was supplemented with position information through position embedding and class information by adding a class token. Because the multi-head attention ^[21]-based transformer encoder has the same input and output dimensions, it can go through several blocks depending on the size of the models (e.g., tiny and small). Multi-head attention is a method of self-attention in parallel by dividing the query, key, and value input by the number of heads. Through this method, the vision transformer identifies the relationships between patches and extracts the image features. Finally, an image classification task is performed through a multi-layer perceptron head after the previous processes.

4.1 Experiment Settings

A vision transformer was trained and validated in four Tesla-V100 GPUs using the Cifar-10 ^[17], Cifar-100 ^[17], and Tiny-ImageNet ^[18] datasets. The Cifar-10 and Cifar-100 datasets have 50,000 training images and 10,000 validation images with 10 and 100 classes, respectively. The Tiny-ImageNet dataset contains 100,000 training images and approximately 10,000 validation images for 200 classes. The most training strategy of DeiT ^[3] and the detailed environmental settings are as follows. Adam ^[31] was used as an optimizer, and set the momentum to 0.9 and weight decay to 0.05. All models were trained for 300 epochs using a batch size of 1,024 and a learning rate of 0.0005. All source codes are referred to the pytorch-based pytorch image models (Timm) library ^[24], and for the 2D-DCT operations, the torchJpeg library ^[25] is used. It should be noted that the DCT block size in all result tables is marked as ``-'' for vanilla vision transformers that do not use the DCT patch embedding.

4.2 Accuracy Evaluation

Table 1 lists the results of applying the proposed method to the vision transformer with the tiny and small models on the Cifar-10 dataset. As suggested, the model was trained by adding discrete cosine transformed images in units of N${\times}$N blocks before performing patch embedding for the vision transformer. When a 4${\times}$4 block size 2D-DCT was adopted on the Cifar-10 dataset, the top-1 accuracy increased by 4.09% and 0.36% for the tiny and small models, respectively. When a 16${\times}$16 block size 2D-DCT was adopted, the top-1 accuracy improved by 4.57% and 3.7% for the tiny and small models, respectively. When the block size was more than 16${\times}$16, the performance was inferior to the others because the detailed information was lost spatially. Therefore, experiments larger than 32${\times}$32 were not performed. In particular, when 2D-DCT was performed with an image size of 224${\times}$224, all spatial features of the image were lost because of the characteristics of 2D-DCT, as shown in Fig. 2(d).

Table 2 shows the same experiment for the Cifar-100 dataset with a tiny model and small model. When the 4${\times}$4 block size 2D-DCT was adopted, the top-1 accuracy increased by 3.86% and 3.59% for the tiny and small models, respectively. When the 16${\times}$16 block size 2D-DCT was adopted, the top-1 accuracy increased by 5.36% and 8.92% for the tiny and small models, respectively.

Experiments were conducted on the Tiny-ImageNet dataset, and a relatively large dataset was used, as shown in Table 3. When the 4${\times}$4 block size 2D-DCT was adopted, the top-1 accuracy increased by 2.92% and 5.37% for the tiny and small models, respectively. When the 16${\times}$16 block size 2D-DCT was adopted, it increased by 3.66% and 5.49% for the tiny and small models, respectively.

As a result, when the 16${\times}$16 block size 2D-DCT patch embedding was applied, performance was increased most in all cases through the proposed method. This result was attributed to the patch being cut into 16x16 during the patch embedding process in DeiT. The increase in computational cost and decrease in speed was negligible. In addition, as the proposed model can be applied directly to most vision transformer models using patch embedding, its compatibility was excellent, making it an easy and general way to improve performance.