2.1 Non-maximum Suppression Hiding

To enhance the detection speed of object detectors in the embedded platforms, this study proposes a parallel processing technique for the convolution operations on the GPU and NMS operations on the CPU, thereby hiding the NMS-processing time into the convolution-processing time. Figs. 1(a) and (b) show the detection process of the conventional algorithm and the process after applying the proposed technique, respectively. As shown in Fig. 1(a), the baseline algorithm does not process the next input image until completing all operations of a currently input image. In autonomous-driving applications, where the images are input through streaming, the processing structure of conventional algorithms is inefficient in terms of hardware utilization. Accordingly, the proposed method employs a pipeline structure to address this issue. As shown in Fig. 1(b), using multi-thread processing, the NMS calculation process of the T frame is hidden by the GPU calculation process of the T+1 frame. Thus, the previous GPU operation result of the T frame stored in the buffer is post-processed simultaneously by the CPU when performing convolution operations of the T+1 frame by the GPU. The processing time of each task is synchronized so that the CPU can process it at the right timing according to the GPU operation. This is possible because there is no data dependency between the GPU inference calculation of the T+1 frame and the CPU post-processing task of the T frame. Therefore, the proposed method is highly efficient in terms of hardware utilization and enables a continuous flow of input images to be efficiently processed.

Fig. 1. Examples of the detection process for the conventional algorithm and after applying the proposed technique.

2.2 Proposed Data Augmentation in Training Step

Among previous augmentation studies, representative Mixup ^[18] and RICAP ^[19] enable a one-stage detector to learn various features based on a square input image, but there is a problem in that the detection accuracy is decreased for non-square image ratios. In addition, fully convolutional network (FCN), which is a typical network structure employed in recent object detectors, determines the amount of computation of the entire network according to the input image size. Therefore, the total computational cost of the FCN is sensitive to the size of the input image; the computational cost of the network decreases with decreasing size of the input image, increasing the detection speed. On the other hand, this leads to a decrease in accuracy. To address this problem, this subsection proposes a new data augmentation technique for autonomous driving embedded systems that enhance detection accuracy without compromising the detection speed.

Fig. 2 shows the proposed data augmentation method using a single image. The example on the left side in Fig. 2 shows data augmentation maintaining the ratio of the original image. In the first step, the original image is cropped randomly. This cropped image is inserted into a square training plate while maintaining its ratio. This process prevents the shapes of the objects in the original image from becoming distorted. In the final step, conventional data augmentation schemes, such as flip, saturation, hue, and exposure changes, are applied ^[20,21]. The example on the right side in Fig. 2 shows data augmentation without maintaining the ratio of the original image. In the first step, the image is cropped randomly, resized into a square, and inserted into a square training plate. In terms of maintaining the original ratio, an affine transform effect for the objects can be obtained using this method. In the final process, the aforementioned conventional augmentation techniques are also applied to produce the final training image. These two augmentation techniques are used during the training phase to enable the CNN to learn various objects, ratios, and features.

Fig. 2. Example of the proposed data augmentation using single image.

Moreover, the use of multiple images rather than a single image can greatly increase the diversity of the data and prevent the overfitting of the CNN with deep layers ^[19]. Fig. 3 shows the proposed data augmentation technique using two images. In Fig. 3, the augmentation processes that maintain and do not maintain the original ratio are the same as in Fig. 2. When two images are used, the square training plate area is divided into two. Each preprocessed input image ($\textit{i.e.}$, image with or without maintaining ratio) is inserted independently into the divided area. The area of the training plate is divided based on the width because the input ratio of the autonomous driving camera is wide-angle. Thus, four training images are generated using two input images, allowing the training data to be expanded. The left side in Fig. 4 shows an example of forming a training plate using two images. The boundary line on the left side in Fig. 4 is not fixed at the center; it can move in the movable direction ($\textit{i.e.}$, up and down), further increasing the number of cases of training data.

Fig. 3. Example of the proposed data augmentation using two images.

Fig. 4. Example of partitioning a training plate according to various images.

A training image is produced using four images to expand the diversity of data beyond two images. As shown in the right example in Fig. 4, images with or without the maintained ratio are inserted in each of the four areas partitioned according to the boundary position and line, thus producing the new training data. The boundary position can move within a specific range in the movable direction ($\textit{i.e.}$, left, right, up, and down), producing more diverse data. Using this proposed augmentation, new global features, which refer to the formation of a new image by combining multiple patches from multiple images ^[19], can be produced to prevent overfitting. Thus, the accuracy is improved significantly.

2.3 Proposed Image Resize in Inference Step

This subsection proposes a preprocessing scheme that enhances the detection speed by eliminating unnecessary operations in object detectors for an autonomous-driving embedded system. In actual autonomous driving, in general, Full HD ($\textit{i.e.}$, 1920${\times}$1080) is used as the input resolution ^[22]. However, because this Full HD size causes serious power consumption and speed degradation, it is common to automatically resize the high-resolution image and process it at a lower resolution. Fig. 5 shows the difference in image resizing with the conventional object detectors and after applying the proposed technique in the inference phase. While most previous methods ^[16] resize the image with a square ratio ($\textit{i.e.}$, Conventional in Fig. 5), YOLO-based object detectors ^[6,13] maintain the actual input ratio when resizing to enhance the accuracy. However, as shown in the Baseline image in Fig. 5, YOLO-based object detectors also execute operations on the square size input by filling in certain pixel values in the margin area in the image ($\textit{i.e.}$, letterbox). In other words, a square input that maintains the input image ratio is generated and processed in the network. Although this provides the advantage of maintaining the input ratio for high accuracy, there is no benefit of reducing the computational cost because the total input still has the same image size of the square ratio.

Fig. 5. Examples of the image resize method in the inference phase on the conventional object detector, baseline algorithm, and proposed scheme.

The proposed inference preprocessing technique can compensate for the inefficient structure by removing unnecessary upper and lower letterbox areas from the baseline method, thereby computing convolution operations only on meaningful pixels and greatly reducing the computational cost. Furthermore, as the algorithm using the proposed data augmentation technique robustly detects diverse input ratios by resizing the image while maintaining the original ratio in the inference phase, the computational cost is reduced while significantly enhancing the detection accuracy. In other words, using the effect of reduced computation, the image can be notably resized again, considering the trade-off between the computing cost and detection speed, thereby improving accuracy.

3.1 Experimental Environment

The superiority of the proposed methods is assessed by experiments using BDD ^[23] and KITTI ^[24] datasets, which are widely used in autonomous-driving research. The same datasets, baseline algorithm ($\textit{i.e.}$, tiny Gaussian YOLOv3), open-source, and experimental settings used in ^[13] are used for a fair comparison.

3.2 Accuracy Evaluation

Table 1 lists the mAP and floating-point operations per second (FLOPs) results of the baseline algorithm ^[13] and the algorithms applying the proposed methods for the BDD and KITTI datasets. The input resolution is set to 512${\times}$512 as in ^[13] to compare the accuracy fairly, excluding the proposed image resize method. It is noteworthy that in BDD, the number of classes is larger than that of KITTI, and the data diversity is also higher. In addition, because BDD applies the strict IoU threshold ($\textit{i.e.}$, IoU>0.75) for all classes in the evaluation, it shows a lower mAP than KITTI ^[24]. When applying the proposed data augmentation process (+Proposed augmentation in Table 1), the mAP is improved by 0.98 pp and 0.96 pp compared to the baseline algorithm for the BDD and KITTI datasets, respectively. It should be noted that as the proposed data augmentation scheme is applied only to the training phase, the computing cost during the inference phase is not increased. Finally, by applying the proposed resize technique maintaining the original image ratio rather than resizing to a square (+Proposed resize in Table 1), the mAP is improved greatly by 1.14 pp for the BDD and 1.34 pp for the KITTI dataset with respect to the baseline. It is noteworthy that applying the proposed resize technique, which removes unnecessary letterbox operations, causes the input image to become smaller than the square size ($\textit{i.e.}$, 512${\times}$512). Accordingly, considering the trade-off between the accuracy and computing cost, it can be scaled up again while maintaining the original image ratio ($\textit{i.e.}$, 672${\times}$384 in BDD and 768${\times}$256 in KITTI. The reason for these values is described in Section 3.3), thereby simultaneously improving the accuracy and reducing the computing cost ($\textit{i.e.}$, FLOPs).

Table 2 lists the accuracy of the existing data augmentation studies ^[18,19] and the proposed techniques. When making a square image size, Mixup ^[18], RICAP ^[19], and the proposed method resize the image into a square using the conventional method shown in Fig. 5 in the inference phase. In contrast, when maintaining the original ratio, they resize the image using the proposed scheme shown in Fig. 5. For square images, all previous techniques, Mixup ^[18] and RICAP ^[19], improve the accuracy compared to the baseline for the BDD and KITTI datasets; however, for input images that maintain the original ratio, their accuracy is reduced greatly. For the KITTI, which comprises wider images than FULL HD, the previous techniques significantly degrade the accuracy compared to the baseline. Even in this case, the proposed method improves the mAP of 0.25 pp for the BDD and 1.85 pp for the KITTI dataset compared to the baseline. In other words, the proposed augmentation method improves the accuracy for both square and original ratio input images. When comparing the accuracy of Mixup ^[18] and RICAP ^[19] for square images with that of the proposed method which maintains the original ratio, the proposed method also shows better result. Nevertheless, the FLOPs of the algorithm applying the proposed method ($\textit{i.e.}$, 8.14${\times}$10$^{9}$ on BDD and 6.19${\times}$10$^{9}$ on KITTI) are smaller than the FLOPs of Mixup ^[18] and RICAP ^[19] ($\textit{i.e.}$, 8.27${\times}$10$^{9}$ on BDD and 8.26${\times}$10$^{9}$ on KITTI), which use the square image size.

3.3 Detection Speed Evaluation

Table 3 shows the inference, post-processing, and total processing times of the baseline ($\textit{i.e.}$, tiny Gaussian YOLOv3 ^[13]) and the algorithm applying proposed schemes in NVIDIA Jetson AGX Xavier ^[25] for the BDD and KITTI datasets. The image size for the proposed resize technique is set to match the FLOPs as closely as possible. Hence, it is set to 672${\times}$384 for the BDD and 768${\times}$256 for the KITTI dataset, while the remaining method is set to 512${\times}$512. Specifically, the size of the original BDD image is 1280${\times}$720, and that of the original KITTI image is 1242${\times}$375. Therefore, the horizontal:vertical ratio is 1.78:1 and 3.3:1, respectively. To match the number of pixels ($\textit{i.e.}$, 262,144 pixels) of the baseline square image ($\textit{i.e.}$, 512${\times}$512), the image size for the proposed resize technique is set to 672${\times}$384 in BDD and 768${\times}$256 in KITTI, while matching the ratio of the original image.

The proposed technique shows greater improvements in performance in default mode, which is typically used in NVIDIA Jetson AGX Xavier. The proposed NMS hiding (+Prop. NMS hiding in Table 3) reduces the total processing time by 7.64 ms for the BDD dataset compared to the baseline. Furthermore, the proposed preprocessing technique (+Prop. Resize in Table 3) reduces the processing time further by 7.9 ms (22.54 %). For the KITTI dataset, applying all proposed techniques reduces the total processing time by 6.57 ms (24.67 %) with respect to the baseline. Finally, applying the proposed schemes to the baseline algorithm improves the accuracy while achieving a detection speed of 36.83 fps (=1000 ms/27.15 ms) for the BDD dataset and 49.85 fps (=1000 ms/20.06 ms) for the KITTI dataset, thereby enabling real-time detection to support faster autonomous driving than the baseline algorithm.