2.1 Key Frame Extraction

There are often a lot of frames in a dance video, and many movements are repeated, which imposes a significant computational burden on dance action recognition. Key frame extraction can be utilized to minimize the computational complexity while enhancing the efficiency of dance movement recognition. Key frame extraction refers to the removal of redundant content in the video and selection of representative image frames that play an essential role in subsequent recognition and classification tasks and that have significant application to video summarization and retrieval ^[12].

The approach used in this paper is based on K-means ^[13], which first requires the selection of appropriate features for effective clustering. The following features are used.

(1) Color: The RGB image of a dance video frame is converted to the hue, saturation, value (HSV) color space ^[14] to extract HSV features. The formulas are:

and

To further reduce the computational effort, HSV color is then synthesized as a one-dimensional vector based on the formula $L=9H+3S+V$.

(2) Texture: In order to extract texture features from a dance video frame, a local binary pattern (LBP) ^[15] is employed. To reduce the computational effort, an LBP equivalent pattern is used to achieve dimensionality reduction processing as follows:

and

where $P$ is the number of sampling points within a circular neighborhood with radius $R$, $b_{i}$ is the pixel value of the pixel point, $b_{c}$ is the pixel value of the center point, and $s\left(x\right)$ is used to determine the value of $b_{i}-b_{c}$.

Based on the above two features, the difference is used to describe the similarity between dance video frames $\mathrm{I}_{\mathrm{c}}$ and $\mathrm{I}_{\mathrm{c}+1}$, which is expressed as follows:

where $D\left(C_{c},C_{c+1}\right)$ denotes the color feature similarity, $D\left(V_{c},V_{c+1}\right)$ is the similarity in textures, and $a_{1}$ and $a_{2}$ are the weights of the two similarities, $a_{1}=a_{2}=0.5$.

Then, using $D\left(I_{c},I_{c+1}\right)$ as the feature, all the dance video frames are classified with the K-means algorithm. After classification, key frames are extracted from each class. For each class, if the number of frames is greater than 8, then the first, middle, and last frames are used as key frames, and if the number of frames is less than or equal to 8, then only the first and last frames are used as key frames.

2.2 Movement Recognition

The multi-feature fusion method presented in this paper is used for movement recognition. First, the features of key frame images are extracted from both spatial and temporal aspects.

(1) Spatial features

In order to better extract information from video frames, the proposed method uses a 3D-CNN ^[16] to extract spatial features from the dance videos. The 3D-CNN obtains spatial features of key frames through 3D convolutional kernels, and the formula can be written as

where $f\left(x\right)$ is the eigenvalue after convolution, $x$ is the pixel value matrix, $w$ is the convolution kernel matrix, and $b$ is the bias.

After convolution, nonlinear processing is achieved from the activation function (the ReLU function here), and the formula is

Then, the feature map obtained in the previous step is compressed through a pooling operation, which can expressed as

where $W_{P}$ is the size of the feature map before pooling, $F_{P}$ is the pooling size, $S_{P}$ is the pooling step length, and $N_{P}$ is the size of the feature map after pooling.

(2) Time features

For temporal features in dance videos, the proposed method uses a long short-term memory (LSTM) network ^[17] for extraction. The LSTM network structure is shown in Fig. 1.

As seen in Fig. 1, LSTM controls the cell state through three gates, where the cell state at the previous moment is $h_{t-1}$, and the input at the current moment is $x_{t}$. The LSTM temporal feature extraction process is shown below.

① The information to be forgotten is calculated through the forgetting gate: $f_{t}=\sigma \left(U_{f}\cdot \left[h_{t-1},x_{t}\right]+c_{f}\right)$.

② The information to be saved is calculated through the input gate: $i_{t}=\sigma \left(U_{i}\cdot \left[h_{t-1},x_{t}\right]+c_{i}\right)$.

③ The cell state is updated with the newly generated information denoted as the candidate information, $\overset{˜}{\mathrm{C}}_{\mathrm{t}}=\tanh \left(\mathrm{U}_{\mathrm{k}}\cdot \left[\mathrm{h}_{\mathrm{t}-1},\mathrm{x}_{\mathrm{t}}\right]+\mathrm{c}_{\mathrm{k}}\right)$, combined to obtain the new cell state: $C_{t}=f_{t}\times C_{t-1}+i_{t}\times \overset{˜}{C}_{t}$.

④ The output value of the current cell state is obtained through the output gate: $o_{t}=\sigma \left(U_{o}\cdot \left[h_{t-1},x_{t}\right]+c_{o}\right)\,,$ $h_{t}=o_{t}\tanh \left(C_{t}\right).$

The parameters in the above equations are presented in Table 1.

Based on the above extracted spatial and temporal features, the process of the multi-feature fusion movement recognition method is shown in Fig. 2.

Fig. 2 shows the steps of this method as follows.

(1) The key frame of a dance video is converted into a five-dimensional tensor (length, width, number of channels, batch, and number of images per frame) that is used as input for the 3D-CNN. After convolution and pooling operations, the spatial feature map is obtained.

(2) The spatial feature map is converted into a 3D tensor (batch, timing step, and the product of pixel matrix length and width) and is the input for the LSTM network to obtain spatial and temporal feature maps of key frames.

(3) The feature map obtained after multi-feature fusion is input for the fully connected layer and the softmax layer. Finally, one-dimensional data are the output for movement recognition in the dance video.

Fig. 1. The LSTM network structure.

Fig. 2. The movement recognition method based on multi-feature fusion.

Table 1. LSTM Parameters and their Meanings.

3.1 Experimental Setup

The experiments were conducted under the TensorFlow framework ^[18] on a Linux system. The change of tensor format is achieved with the reshape function, and 3${\times}$3 convolution is used for the convolution kernel of the first two layers of the 3D-CNN, with 1${\times}$1 convolution used for the last two layers. Max-pooling is used, and the Adam optimization algorithm is used for training the network at a learning rate of 0.001.

There were two kinds of experimental data. The first was the DanceDB dataset ^[19], which contained 48 dance videos involving 12 dance movements. Each movement was labeled with an emotion tag (e.g., scared, angry). The frame rate of the videos was 20 frames per second, and the size of each frame was 480${\times}$360.

Second was a self-built dataset containing 96 dance videos recorded by six students majoring in dance, all in the same context. These videos included three types of dance, each with complex and variable movements. The frame rate of the videos was also 30 FPS, and again, the size of each frame was 480${\times}$360. Some of the frames are in Fig. 3.

All the videos had key frames marked by professional dance teachers, and the following indicators were selected to assess the effects of key frame extraction.

(1) Recall ratio: the number of key frames correctly extracted, divided by the number of key frames correctly extracted plus the number of missed frames.

(2) Precision ratio: the number of key frames correctly extracted, divided by the number of key frames correctly extracted plus the number of frames falsely detected.

(3) Deletion factor: the number of frames falsely detected, divided by the number of key frames correctly extracted.

The effectiveness of movement recognition was evaluated using accuracy: the number of videos correctly recognized divided by the total number of videos.

Fig. 3. Example dance video frames from the self-built dataset.

3.2 Analysis of Results

First, the key frame extraction method from this paper was compared with the following two methods:

① The color feature-based approach proposed by Jadhav and Jadhav ^[20], and

② The scale-invariant feature transform approach proposed by Hannane et al. ^[21].

The key frame extraction results of the three methods are shown in Table 2.

From Table 2, we see that, first, the recall ratios of the methods proposed by Jadhav and Jadhav and Hannane et al. were below 80% for the DanceDB dataset, while the recall ratio of the proposed multi-feature fusion method was 82.27% (10.82% higher than Jadhav and Jadhav, and 8.81% higher than Hannane et al.). Secondly, the precision ratios of the Jadhav and Jadhav and Hannane et al. methods were below 70%, while the precision ratio from multi-feature fusion was 72.84% (5.97% higher than Jadhav and Jadhav and 4.65% higher than Hannane et al.). Third, the deletion factor of the multi-feature fusion method on the DanceDB dataset was 3.01, which was 1.41 lower than the method by Jadhav and Jadhav, and 0.55 lower than the method by Hannane et al.

The recall and precision ratios of all the methods when processing the self-built dataset improved to some extent, compared to the DanceDB dataset, and the deletion factors were also smaller, which may be due to the relatively small number of dance types in the self-built dataset. In comparison, the recall rate of the multi-feature fusion method was 84.82%, and the precision ratio was 81.07%, both of which are higher than 80% and significantly higher than the other two methods. The deletion factor with multi-feature fusion was 2.25, which was significantly smaller than the other two methods. Based on the results with the two datasets, the multi-feature fusion method had fewer cases of missed and incorrect detections of key frames, as well as better extraction results.

Taking the dance called Searching as an example, the key frames output by all three methods are presented in Fig. 4. More key frames were extracted by the Jadhav and Jadhav and the Hannane et al. methods, but some frames are not clear. Key frames extracted by the multi-feature fusion method gave a complete description of the movement changes in the dance video, a good overview of the video, and the extracted movements are clear. Therefore, the multi-feature fusion method can be used to provide movement recognition services.

Based on the key frame extractions, the results of movement recognition were analyzed to compare the effects of different features and different classifiers on the results of dance video movement recognition. The compared methods are

Method 1: using only spatial features plus the softmax classifier,

Method 2: using only temporal features plus the softmax classifier, and

Method 3: using spatio-temporal features plus the SVM classifier ^[22].

Comparison of accuracy from the above methods and the multi-feature fusion method is presented in Fig. 5.

According to Fig. 5, movement recognition accuracy was low in all cases when only one feature was used. The multi-feature fusion method exhibited accuracy of 42.67% with the DanceDB dataset, which is 11% higher than Method 1 and 8.71% higher than Method 2. For the self-built dataset, multi-feature fusion had an accuracy of 50.64%, which is 17.26% higher than Method 1 and 14.72% higher than Method 2. This indicates that the extracted features could have some influence on movement recognition when using the same classifier. The comparison shows that using only spatial features or only temporal features led to a decrease in recognition accuracy, while the combination of spatio-temporal features produced better recognition of dance video movements.

Comparing Method 3 and the multi-feature fusion method, the difference in classifiers resulted in differences in accuracy. Recognition accuracy from Method 3 on the DanceDB dataset was 38.56% (4.11% lower than the method proposed in this paper), and with the self-built dataset, recognition accuracy from Method 3 was 40.11% (10.53% lower than the proposed method). This indicates that the softmax classifier provided better recognition of different dance movements than the SVM classifier. The SVM required a lot of computation time for multi-classification recognition, and the selection of the kernel function and parameters depended on manual experience, which is somewhat arbitrary. Therefore, the softmax classifier was more reliable.

A movement identification approach based on trajectory feature fusion was proposed by Megrhi et al. ^[23]. It was compared with the proposed multi-feature fusion method, and the results are presented in Fig. 6.

Fig. 6 shows that movement recognition accuracy from the method proposed in this paper was significantly higher for the two dance video datasets. The reason the accuracy of both methods was higher with the self-built dataset is that it included more dance movements. For DanceDB, the recognition accuracy of the Megrhi et al. method was 39.52% (3.15% lower than multi-feature fusion), and for the self-built dataset, recognition accuracy of the Megrhi et al. method was 46.62% (4.02% lower than multi-feature fusion). These results demonstrated the multi-feature fusion method is effective in identifying movements from dance videos.

The recognition accuracies of the method proposed by Megrhi et al. and the multi-feature fusion method for different dance movements in the self-built set were further analyzed, and the results are shown in Fig. 7.

Fig. 7 shows that accuracy from the Megrhi et al. method was below 50% for all three dances, among which the lowest accuracy (45.12%) was with Green Silk Gauze Skirt, and the highest accuracy (47.96%) was with Memories of the South. Compared with the Megrhi et al. method, recognition accuracy from multi-feature fusion was 5%, 2.55%, and 4.51% higher for the three dances, which proves the reliability of the multi-feature fusion method in recognizing different types of dance movement.

Fig. 4. Comparison of key frame extraction results.

Fig. 5. Comparison of accuracy from dance video movement recognition.

Fig. 6. Comparison of recognition accuracy from multi-feature fusion and trajectory feature fusion.

Fig. 7. Accuracy comparison with the self-built set.