1. Introduction

Recently, the mobile processor manufacturer, Qualcomm, introduced 8K (kilo) UHD (Ultra High Resolution) video recording ability to their chipset ^[1]. The chipset can process 24 frames of 8K video per second, thereby allowing the creation and consumption of UHD videos for the general public. In addition, captured UHD video can be broadcast from a mobile phone to the internet in real-time using 5G (fifth-generation) communication service ^[2-5]. A study by ETRI (Electronics and Telecommunications Research Institute, Korea) forecasted that the demand to consume 8K UHD contents would increase as more 8K UHD display panels are supplied in 2025 ^[6,7].

The frame from a live stream of UHD video can be utilized in various applications using image-processing techniques. One of the major areas that use real-time video to detect information is lane or object detection of vehicle-mounted video recorders to alert drivers to hazardous situations. The video frame can also be utilized to detect artifacts of a PCB (Printed Circuit Board) during the soldering process ^[8], and to detect fire from a surveillance camera ^[9]. Moreover, many conventional AR (Augmented Reality) applications analyze a frame-from-video to calibrate a camera or detect simple object information ^[10,11].

A captured video is also utilized in many deep learning applications combined with DNNs (Deep Neural Network), RNNs (Recurrent Neural Network), or CNNs (Convolution Neural Network) ^[12]. The application includes sophisticated object segmentation, human emotion recognition, situation awareness, and video analysis ^[13-17]. Moreover, several efforts have been made to process video with neural networks on mobile devices because recent mobile processors have dedicated NPU (Neural Processing Unit) or AI (Artificial Intelligence) accelerators.

One major time-consuming part of introduced deep learning-based methods is the pre-processing of the video frame. The pre-processing includes conventional image processing, e.g., color scale conversion, thresholding (binarization), and convolutional filtering. Fig. 1 gives an example of image pre-processing used widely for neural networks.

To gain as much information as possible, a wide-angle high-resolution camera is used to capture video, which leads to longer pre-processing times. Therefore, minimizing the pre-processing time for a high-resolution video frame is an important factor for increasing the efficiency of a deep learning process.

A Sobel operator is one of the most used convolutional image pre-processing methods. Sobel operators detect edge information and are used widely in classifications [11, 18-20]. A de-facto computer vision library OpenCV (Open Computer Vision) also provides image pre-processing algorithms, including the Sobel operator ^[21]. OpenCV provides a good general solution for image processing algorithms, but it lacks optimization for specific functions owing to its generality.

This paper introduced an optimized image pre-processing method, specialized in image convolution for videos with very high resolution, using OpenCL (Open Computing Library) ^[22]. The proposed scheme can fully utilize the resources of embedded systems with low-end mobile processors.

The remainder of this paper is organized as follows. The related works section introduces previous research to accelerate image convolution. In optimizing the OpenCL kernel architecture scheme section, this paper introduces an optimizing technique to process a convolution kernel. The performance evaluation section included a prototype video player implementation. In addition, the performance is evaluated, and a comparison is made with the well-known image processing library. Finally, the last section concludes the paper with discussions and effects of the proposed scheme.

Fig. 1. Examples of image pre-processing methods for neural networks.

2. Related Works

Several efforts have been made to accelerate image processing using OpenCL. The latest research ^[23] shows that it can process Sobel operation for a 2K image in 0.926 milliseconds. On the other hand, the research platform is based on a desktop platform rather than on mobile platforms. Table 1 lists the performance specifications of Nvidia GTX1060, which was used in a previous study ^[23], and ARM Mali-T880, which was used in this paper.

Although both GPUs (Graphics Processing Units) were launched in the same year, the performance gap between the two chipsets was apparent because of the limited power consumption of the mobile architectures. Therefore, another approach to accelerate image processing for mobile architectures is required.

Another study ^[24] optimized an image process by manipulating data to achieve the minimum load and store time. The research achieved an up to 3.3 times faster processing time than the conventional Sobel operation using OpenCL on mobile GPU. The data-level optimization technique is a good solution in accelerating parallel tasks.

Because the study goal was to process a video frame while playing real-time video, it is important to consider that the GPU resources are already being used to decode a video. The performance will deteriorate if more resources are used to manipulate data due to frequent context switching between different tasks.

A parallel library provided by OpenCV also provides a good general solution for convolution, including the Sobel operator. The library was also implemented with OpenCL to utilize parallel processing, but the OpenCV library does not consider its running platforms. Additionally, conversion from a ffmpeg decoded YV12 image type to an OpenCV RGB image type should be conducted to process an image, which requires multiple sequential reads of the memory, leading to performance reduction ^[25].

3. Optimizing OpenCL Kernel Architecture Scheme

OpenCL was used for parallel processing API (Application Programming Interface) because this study aimed to perform real-time image processing while playing a real-time video for mobile architectures. The Exynos 8890 processor was also targeted because of its generality in being embedded in the Samsung Galaxy S7. The chipset has an integrated GPU (Graphics Processing Unit), ARM Mali-T880, which supports a maximum work-group size of $\textit{(256, 256, 256)}$.

In the scheme, each pixel in each thread is processed. A thread is the minimum size of a program to run kernel source code on a single GPU core. The multiple threads can be launched physically and simultaneously with the size of the work-group. Multiple work-groups can be logically contructed and simultaneously launched in the GPU. They are typically arranged into the ND-Range (n-dimensional range) data types in OpenCL. Therefore, although the total pixel number of 8K $\textit{(7680 * 4320)}$ video is larger than the maximum work-group size of the chipset, the threads can be divided into multiple work-groups within the size of the ND-Range.

A work-group size should be adjusted adaptively, so it does not exceed its maximum work-group size while approaching the maximum size. Because launching OpenCL kernels to a power of 2 is the most ideal, it is important to calculate the total number of threads $\textit{N}_{threads}$ using the equation

where $\textit{Width}_{f}$ and $\textit{Height}_{f}$ represent the width and height for the video frame. With $\textit{N}_{threads}$, the optimal adoptive work-group size $\textit{(X, Y, Z)}$ can then be calculated with the procedure in Fig. 2.

The convolution kernel used was the Sobel kernel, as shown in Fig. 3. A general solution could be derived for all 3*3 convolution kernels by requiring the kernel as a parameter. In the proposed prototype, however, a fixed kernel was used for the best performance. Although fixed kernels lack generality in the programming aspect, the kernel itself can be adjusted if other kernels are needed.

OpenCL programs can access the texture memory area directly, while other CPU-based programs should copy the image data to the main memory area. Thus, in the case of OpenCL, there is no need to convert image color space for use in OpenCL kernels, and the image type conversion overhead can be minimized.

Fig. 2. Adaptive work-group size calculating procedure.

Fig. 3. Sobel convolution kernels.

Fig. 4. Flow charts of each thread for prototype video player implementation.

4. Performance Evaluation

If a commercial player is used to process frames, multiple unnecessary image conversions from YV12 color space to RGB color space should be conducted. Therefore, a simple video player was implemented to evaluate the performance of the proposed OpenCL kernel. The video player consisted of two main threads, a decoder thread, and an image processor thread.

The decoder thread decodes a video file until it reaches the EOF (end of file) and queues it to the global queue. The decoder is implemented using the ffmpeg library, and it fully supports the hardware acceleration of the GPU for video decoding. The frame processor thread obtains a decoded frame from the queue and launches the OpenCL kernel or OpenCV function to process the frame. Fig. 4 shows flowcharts of each thread for prototype video player implementation.

Using the proposed prototype implementation, the frame-processing performance was measured and compared with that in the OpenCV library. HEVC (High Efficient Video Codec) and H264 codec were used in the experiment. Although both codecs require high computing resources, they are used widely to minimize their file capacity. Only HEVC is used for 8K video. Table 2 lists the video resolution, codec, and bitrate information used for the experiment. The playback time of each video is the same.

Table 3 lists a performance for each resolution, codec, and method. The analysis was conducted using the Exynos 8890 processor. The table includes the decoding and image convolution time for each resolution, codecs, and methods. The result showed that the proposed method performs better than OpenCV for all resolutions and codecs. The proposed scheme can handle a single frame of 8K video at least in one second, even though it requires context switching because it exceeds the maximum work-group size. A higher frame rate can be expected if the decoding speed of the video can be minimized with a state-of-art chipset.

Another noticeable performance is the frame decoding time for each method. The decoding time was faster when OpenCV was used for convolution than when OpenCL was used. The decoding performance gap occurred because of the higher usage of GPU resources for the proposed scheme as it uses all the available physical threads of the GPU. On the other hand, the overall time consumption for each video frame processing was always smaller in the proposed scheme.

5. Conclusion

An optimized OpenCL kernel scheme that can perform convolution for video frames, including 8K UHD video was designed. The proposed design achieved better overall performance than the well-known conventional image-processing library OpenCV. The performance gap occurred because the OpenCV parallel algorithms focused on generalizing overall image processing algorithms, whereas the current design focused on specialized kernels for convolution. On the other hand, the video decoding time of the proposed scheme was longer than when OpenCV was used. The reason for the difference in decoding time was that the scheme fully utilizes the GPU resource. Thus, the available decoding resource of GPU was smaller.

In the future, the proposed scheme is expected to be adopted to a processor that can decode 8K HEVC video in real-time. In addition, the image processing workload is expected to be distributed between the central prosceiing unit and GPU to achieve higher performance. Furthermore, a solution for video with more than 8K resolution should be considered because it exceeds the maximum work-group size of the chipset.