2.1 TensorFlow

TensorFlow is an open-source software library developed by Google ^[1], which performs numerical computations based on data flow graphs to conduct machine learning and deep neural network research ^[1]. TensorFlow provides high-level APIs that make it easy to construct a neural network with high flexibility that can deploy computations either on CPUs or GPUs on desktops, servers, or mobile devices. The basic unit of TensorFlow is a computational graph where each node represents an operation, and each edge, called a tensor, represents data. The data comprise a set of primitive values shaped into an array of any dimension. The computation node takes zero or more tensors as input and produces a tensor as output.

The CNN is the current state-of-the-art image classification architecture, and it applies a series of convolution filters to raw images in order to extract and learn higher-level features. The CNN receives input and transforms it through a series of hidden layers composed of convolutional, pooling, and dense fully connected layers. The convolutional layer is the core of the CNN and consumes most of the computational capacity from among all the layers. TensorFlow employs a few convolution algorithms and selects one of them to achieve optimal performance in terms of memory usage and execution time with respect to the input parameters. The selection is performed in the profiling phase.

2.2 Profiling

Algorithm 1 presents TensorFlow's convolution profiling algorithm. TensorFlow examines whether each convolution node was profiled when triggering the node. If the node was profiled, the selected algorithms are chosen. Otherwise, profiling is performed to find the best algorithm for the node.

There are two reasons for profiling to enhance performance: identifying an algorithm to minimize memory usage and one to minimize execution time. To find the algorithms, all available convolutions are executed, depending on the convolution type, such as forward, backward input, and backward filter. Table 1 shows the available algorithms from TensorFlow with NVidia GPUs. In the forward convolution type, IMPLICIT_GEMM, IMPLICIT_PRECOMP_GEMM, GEMM, and FFT algorithms are available. All the algorithms perform a matrix-to-matrix product, but their memory usage and execution times differ. An implicit algorithm like IMPLICIT_GEMM requires less memory, but executes for a longer time than explicit ones, such as GEMM and FFT ^[23,25].

Fig. 1 shows an example record from profiling. The first four lines represent the convolution node's ID (name: starting address), an applied algorithm number, and its execution time in $ms$. Line 5 shows the convolution type. In this example, the type is $\textit{backward_filter}$, and the algorithm numbers on lines 1 to 4 represent the algorithms in Table 1, in order. In other words, algorithm numbers 0, 1, 2, and 3 imply $\textit{ALGO_0}$, $\textit{ALGO_1}$, $\textit{FFT}$, and $\textit{ALGO_3}$, respectively. The fields in the input and output shapes in lines 6 and 8, respectively, present the data type (F32: 32-bit floating-point), batch size, image width, image height, and image channel. For the filter shape in line 7, the fields are the filter width, filter height, input channel, and output channel. Finally, the last line shows the two selected algorithms from the profile: the first (3), achieves the minimum execution time, and the second (0) requires the least memory.

Basically, the profiled information includes convolution type, input shape, filter shape, and output shape. We show in Section 3.1 that the batch size has no significant effect on algorithm selection, so we can use the profiled information from the first run for future runs.

Algorithm 1. TensorFlow’s convolution profiling.

Fig. 1. Example record from TensorFlow profiling.

2.3 Profiling Drawbacks

Fig. 2 presents the memory usage of our GPU platform (detailed in Section 4) over time in TensorFlow when the batch sizes (the numbers of processed images in one execution) are 16 and 32 for the Inception-V3 application, which is a widely used image classification model trained with the ImageNet Large Visual Recognition Challenge 2012 datasets. From the experiment, we identified drawbacks in profiling execution. Memory usage overshoots approximately 10 seconds after the start of execution, and the overshoot was more than 70% of the total memory, after which memory usage was constant at less than 40\%.

We found that the overshoot resulted from memory allocation at the beginning of program execution, i.e., at the profiling phase. The memory required for profiling all the algorithms was allocated initially. The allocated memory for the selected algorithm was used for the future execution, and the other regions were deallocated. More memory for processing more batches cannot be allocated for future runs of the selected algorithm. Therefore, profiling limits the exploitation of higher data-level parallelism.

Fig. 2. Memory usage in GPUs over time for the Inception-V3 application in TensorFlow.

3.1 Algorithm Selection

Table 2 compares profiling information for minimum execution time while running Inception-V3 on batches of 16 and 32. We categorize the results into three cases: $\textit{Case~1} $represents two batches for which the same algorithm was selected; $\textit{Case 2}$ represents a situation with the same number of different algorithm selections; $\textit{Case 3}$ represents a situation with different numbers of the same algorithm being selected. In this example, $\textit{Case 1} covers 90% of the total selections, and the other cases cover the remaining 10\%.

In $\textit{Case 1}$, there is no difference between, and no obstacles to storing and reusing, the results, but $\textit{Case 2}$ and $\textit{Case 3}$ require careful approaches. In Fig. 1, which shows the profile information, algorithm 3 was selected for minimum execution time. However, the execution times of algorithm 3 and algorithm 1 are 0.809984 $ms$ and 0.816864 $ms$, respectively, a difference of only 0.8%. Between $\textit{Case 2}$ and $\textit{Case 3}$, the execution time difference is negligible, and therefore, there is no significant difference in the overall performance when using either algorithm. In the case of algorithms with minimum memory usage, the profiles are always the same.

Table 3 presents some comparison results of profile information obtained by running the algorithm twice with a batch size of 32. We compared the results to confirm there were no obstacles in storing and reusing the information when there is repeated execution under the same conditions. As a result, most of them belonged to $\textit{Case 1}$, and the remaining belonged to $\textit{Case 3}$, but there was no problem in storing and reusing them, because the performance of the two algorithms does not differ noticeably, as mentioned above. Therefore, the profile information can be saved to the profile record, and the profile record can be reused for four classification criteria: convolution type, input shape, filter shape, and output shape (excluding batch size).

3.2 Overall Architecture

Algorithm 2 shows our proposed convolution profiling algorithm. For each convolution node, the algorithm examines whether the convolution node was profiled. If the node was not profiled, it checks whether a profile record is available from the previous run. If a profile record is available, we select the algorithm from the profile record. Otherwise, we perform profiling to find the best algorithm, and save the profile information in the profile record file for future runs.

Algorithm 2. Our proposed convolution profiling.