3.1 Image Preprocessing

According to the characteristics of a defect image, the defect part of the image cannot help us understand the content of the image. Therefore, when research measure the similarity of the image, research do not consider the similarity of the image, and the similarity of the overall content of the image is determined by the similarity of the information contained in the display part of the image. Image segmentation is basic work in the field of image processing and is also an important step in the following image processing and analysis. Thus, before extracting the low-level features of the image, in order to improve the effect of image annotation, all the images to be annotated are divided into regions and segmented uniformly.

In region division and image segmentation, the method of region selection was used. The aim of region selection is to select several regions from the whole image in a certain way, describe the content of the image based on these basic regions, and better mine the information of different objects in the image ^[12]. Image region selection methods are mainly divided into fixed division, image segmentation, and prominent points. Among these, image segmentation is the most effective and the longest method of region selection. Image segmentation aims to segment the image into regions corresponding to several objects so that each region corresponds to one object.

<New paragraph> Image segmentation is basic research content in the field of computer vision. The regions after image segmentation are often irregular regions. A regional covariance description method can be used to extract each region. Let I be a 1-D gray or 3-D color image and F be a feature image of W * H * d extracted from I:

$\left(x,y\right)$ is the coordinates of the feature point, W is the width of the image, H is the height of the image, d is the number of features extracted, and $\phi \left(\cdot \right)$ can be any mapping function, such as the image gray value, color, gradient, and filter response. For a given region R, the order $z_{i}$ is the d-dimension characteristic point inside R. Region R is represented by the covariance of the feature points:

$\mu =\frac{1}{n}\sum _{i=1}^{n}z_{i}$ is the mean of all feature points, and n is the number of pixels in the region.

Firstly, an image is segmented into different regions to make each region correspond to an object. Then, the region after image segmentation is described using covariance. The difference from the original covariance description is that the region corresponding to the original covariance is a regular rectangular region, while the region corresponding to the covariance matrix is an irregular region. Regular regions usually contain multiple objects, and the region after image segmentation usually corresponds to a semantic object. Covariance description is a regional representation method from the perspective of regional feature point distribution, which is independent of the size of the region. For the same semantic goal, the corresponding regional distribution will be similar, and the covariance will also be similar. Obviously, the region after image segmentation is represented by covariance, which can be used to distinguish regions of different objects.

3.2 Image Feature Extraction

3.3 Similarity Measure of Image Features

In the set of segmented sub-blocks with annotation, the similarity of the tagged image is measured. That is, $I=\left\{I_{1},I_{2},\cdots ,I_{s}\right\}$ of the segmented sub-blocks with annotation is used to obtain nearest neighbor $I_{i}=\left\{I_{1}^{i},I_{2}^{i},\cdots ,I_{s}^{i}\right\}$. Each image segmentation block with annotation is represented as $K$, and then all the image segmentation blocks with annotation in the training set are represented by matrix $I'=\left\{I_{1},I_{2},\cdots ,I_{i}\right\}\in R^{1\times si}$. Because each part of the image has certain spatial information, each block of the image to be labeled is located in the training set. That is, only the subset $I_{1}^{i}$ of the training set is taken into account when the nearest neighbor $I_{1}$ is obtained. For the lower-level eigenvectors extracted from the two partitioned blocks, a and b are represented as $F_{a}\left(hsv_{1}^{a},hsv_{2}^{a},\cdots ,hsv_{256}^{a},tex_{1}^{a},tex_{2}^{a},\cdots ,tex_{t}^{a}\right)$ and $F_{b}\left(hsv_{1}^{b},hsv_{2}^{b},\cdots ,hsv_{256}^{b},tex_{1}^{b},tex_{2}^{b},\cdots ,tex_{t}^{b}\right)$, respectively. The distance between the two partitioned blocks is:

Then, Research can find $D_{s}=\left\{d_{s1},d_{s2},\cdots ,d_{sk}\right\}$ of the distance vector between $K$ and $I_{s}$, and $d_{s1}<d_{s2}<\cdots <d_{sk}$. $d_{s1}$ is the distance between two blocks with the highest similarity <note: awkward/ambiguous>. Matrix $D=\left[D_{1},D_{2},\cdots ,D_{s}\right]\in R^{1\times sk}$ is the distance between all partition blocks of the incomplete image to be marked and their corresponding neighborhood blocks ^[13,14]. <note: ambiguous (incomplete)> Research define the subblock and subblock similarity metrics as:

$d_{ab}$ denotes the distance between partition block a of the incomplete image and its nearest neighbor block b. The closer the distance is between two blocks, the greater the similarity measure is. The similarity measure between segmented sub-block $I_{s}$ of the incomplete image to be labeled and its corresponding neighborhood segmented sub-block is $W_{s}\,\left(w_{s1},w_{s2},\cdots ,w_{sk}\right)\,,$ and the similarity measure matrix of $I$ is $W=\left[W_{1},W_{2},\cdots ,W_{s}\right]\in R^{1\times sk}$.

3.4 Training of Models

The training process for the model is shown in Fig. 1. In the training process, the input of the model is the extended training set of the sentence description generated by the model:

Here, $m$ ranges from 1 to $M$, and $C_{\theta c}\left(I\right)$ represents the output of the last layer of $CNN$. $h_{m}$ is the output of the hidden layer, $h_{0}$ is initialized as a 0 vector, and the input of the neurons in the hidden layer includes the expanded word vector $x_{m}$ and the previous moment’s information $h_{m-1}$ (contextual information). But research only consider the influence of the image information $b_{v}$ in the first step of training. Experiments have shown that the effect is better than adding $b_{v}$ at each step, $x_{1}$ as a specific START vector, $x_{2}$ as the first word in a sentence, $x_{3}$ as the second word, and $x_{m}$ as the last word.

<New paragraph> $y_{m}$ is the output of the output layer, indicating the probability of a word in the dictionary and the probability of a terminator. The $y_{1}$ tag corresponds to the first word in the sentence during the exercise, the $y_{2}$ tag corresponds to the second word, and the $y_{M}$ tag being a specific END vector. The training of the model is then realized.

Fig. 1. Training process for a model.}

3.5 Image Annotation Algorithm

Regarding the incomplete image, the incomplete part and the display part of image information have relevance <note: ambiguous>. To eliminate the interference of the defective part on the image recognition, research use a certain spatial relationship between the subblocks of the image display part. This kind of spatial relation is more embodied between the partitioned sub-blocks and the partitioned sub-blocks of the image than between the objects and objects in the image. In view of this characteristic, in the process of automatic labeling of incomplete image, the fused spatial information is the proportion of the segmented sub-blocks of the image in the whole image spatial distribution (that is, the spatial structure information).

According to the idea of similarity transfer, the similarity between tagged words and images is related. The higher the similarity between images is, the closer their tagged words should be. Therefore, the similarity between images can be transferred to the relevance between their corresponding tagged words. The similarity measure between image blocks is used to transfer the similarity relationship between images in the process of annotation and to transform the similarity from an image to annotated words. The similarity measure transfer matrix defining image I is $W^{*}$:

In this equation, $f\left(d\right)$ is the transfer function of the similarity measure.

In this paper, all the annotated words corresponding to the partition block $I_{i}(i=1,2,\ldots ,9)$ of unknown annotated image I are represented by annotated word vector $T(t_{1},t_{2},\ldots ,t_{p})$. p is the total number of annotated words, and repeated annotated words are considered. For the nearest neighbor block j of the sub-block i of image I, the corresponding annotation word is marked as 1 in T, and no corresponding annotation word is marked as 0. <note: ambiguous> $T_{'}$ is obtained. Research multiply $T_{'}$ and the corresponding similarity measure transition value $w_{ij}^{*}$ to find the similarity measure transition vector of annotation words:

For the complete image,

$M^{*}$ is the transfer vector of similarity measure corresponding to the annotation words of the image. In order to make the data format consistent <note: ambiguous>, $M^{*}$ is normalized, and then the threshold is set according to the actual situation, and the annotation words above the threshold are reserved. Thus, the automatic labeling of the incomplete image is realized. During the annotation, research set the iterative step $r_{t}=r_{0}/(1+0.001\eta t),\,\,\eta =10\mathrm{e}-5.$ The sub-concept parameter K = 5 of the label should not be too large. In addition, when the K value is too large, the training time of the algorithm will be increased. The overall algorithm flow is:

(1) Training process:

Input: Training dataset $\left\{\left(B_{1},Y_{1}\right),\left(B_{2},Y_{2}\right),\cdots ,\left(B_{M},Y_{M}\right)\right\}$, parameters $K$ and $\gamma _{t}$ output: $u_{lk},V_{lk}\left(l=1,2,\cdots ,L+1;\right.$ $\left.k=1,2,\cdots ,K\right)$

1) Training.

2) Initialize $u_{lk}$ and $V_{lk}\left(l=1,2,\cdots ,L+1;k=1,2,\cdots ,K\right)$.

3) Circular execution.

The training images to be labeled are divided into regions and segmented, and the segmented image features are extracted to measure the similarity of the images.

(a) A package B and one of its associated tags is randomly selected.

(b) Research obtain key examples and their corres-ponding sub-concepts: $\left(X,k\right)=\arg \max _{X\in B,k\in \left\{1,\cdots ,K\right\}}f_{yk}\left(X\right)$.

(c) If $y$ is not a virtual label $\hat{y}$, then $\overline{Y}=\overline{Y}\cup \left\{\hat{y}\right\}$.

(d) Circular implementation: $i=1\colon \left| \overline{Y}\right| $.

(e) Random selection of an unrelated tag $\overline{y}$ from the $\overline{Y}$ and selection of the key example $\overline{X}$ and its corresponding sub-concept $\overline{k}\colon \left(\overline{X},\overline{k}\right)=\arg \max _{\overline{X}\in B,\overline{k}\in \left\{1,\cdots ,K\right\}}f_{\overline{y}\overline{k}}\left(\overline{X}\right)$.

(f) If $f_{y}\left(X\right)-1<f_{\overline{y}}\left(X\right)$.

(g) Order $v=i$.

(h) Updating and standardizing $u_{yk},v_{yk},u_{\overline{y}\overline{k}},v_{\overline{y}\overline{k}}$.

4)~Ending the cycle: The cycle ends when the termination conditions are met.

(2) Testing stage:

The associated label set for the test package B test is: $\left\{l\left| 1+f_{l}\left(B_{test}\right)>f_{\hat{y}}\right.\left(B_{test}\right)\right\}$.

4.1 Experimental Conditions

The software and hardware environment during the experiment is as follows:

Automatic image annotation datasets are mainly divided into a training set, verification set, and test set.

(1) Training set

The Image Auto-Tagging Training Set is a Corel5K dataset (https://dumps.wikimedia.org/enwiki/latest/) con-taining 82,000 images, each with varying degrees of mutilation <note: awkward/ambiguous> and five sentences of a description generated by Amazon’s Mechanical Turk service.

(2) Verification set

For the Corel5K dataset, 500 secondary images from the training set are selected as the validation set.

(3) Test set

The test set used for image annotation in this paper was 2500 images selected from the training set.

4.2 Experimental Results

4.2.1 Partial Image Annotation Results

Some of the experimental results of the designed automatic labeling algorithm are shown in Fig. 2. <note: Paragraphs should generally be at least 3 sentences> It can be seen from Fig. 2 that the proposed algorithm can be used to label the incomplete image effectively, and it does not matter if the image is cropped, covered by foreign objects, or blurred in a large area. This is because the algorithm designed in this study can be divided into regions, which is conducive to the extraction of image features, after extracting features to improve the accuracy of image annotation.

Fig. 2. Partial labeling results.

4.2.2 Recall Ratio

In the Corel5K dataset, 3000 different types of images were randomly selected as subjects. By comparing the completeness of image labeling between this method and other methods ^[7,8], the advantages of this method were verified. The results are shown in Fig. 3. From Fig. 3, Research can see that the image label of the incomplete image has integrity <note: awkward/ambiguous>, and the recall rate is 97%, which is obviously higher than the other two methods, so our method can effectively identify the image and has strong practicability.

Fig. 3. Label search results.}

4.2.3 Precision Rate

The standard deviation value $\sigma $ of the normal distribution in formula (4) is obtained by iteration using:

(23)

$ \sigma =\frac{H}{K}\times 100\mathrm{\% } $

$H$ is the number of labels marked, $K$ is the total number of labels. The smaller the $\sigma $ value is, the clearer the image is, the more accurate the result is, and the higher the value of the method is.

Fig. 4 shows that after 300 iterations, the standard deviation of the normal distribution tends to be stable. The $\sigma $ value is 11 when the incomplete image is labeled automatically by this method. The $\sigma $ value is 18 and 19 when the incomplete image is labeled by methods of Literature ^[7] and ^[8]. Compared with the results of the other methods, the precision of the proposed method is higher because it constructs a Gaussian pyramid, which can obtain the logarithm of the dimension of the image at the top of the pyramid structure, which reduces the standard deviation of the normal distribution and improves the precision.

The Peak Signal Noise Ratio (PSNR) was used as an indicator to evaluate the performance of the methods proposed in the literature, as shown in Fig. 5. In the process of algorithm iteration, the PSNR of the three algorithms increased. At 200 iterations, the PSNR value of the proposed method is 15.76. The PSNR value of one method in the literature ^[7] is 15.05, and the PSNR value of the other literature method ^[8] <note: It seems that one of these should be ``^[8]''; clarification by the author is needed> is 14.15. It can be seen that the PSNR value of the algorithm proposed in this study is obviously superior to those of the other two algorithms, which indicates that the performance is better.

Fig. 6 shows the F1 value changes of the methods in this paper, a method in the literature ^[7], and another method in the literature ^[8] during the iteration process. It can be seen that when the number of iterations increases, the F1 value of several models also increases. This shows that in the process of iterative training, several models fit the data, and the fitting effect is improved. When the F1 value increases to a certain extent, it will not increase more, and the F1 value curve also tends to be stable. This is because after enough iterative training, the model has reached its best performance, and the accuracy of the model can no longer be improved.

<New paragraph> It can be seen that the method in this paper achieves the best accuracy when it has 164 iterations, while a method in the literature ^[7] needs 172 iterations, which is 8 times more than the method in this paper. However, the other method ^[8] needs 214 iterations, which is 50 times more than that in this paper. After reaching the best performance, the F1 value of the method in this paper is 0.95, while the F1 value of a method in the literature ^[7] is 0.92, which is 0.03 lower than the method in this paper. The F1 value of the other method ^[8] is 0.89, which is 0.06 lower than the method in this paper.

Fig. 4. Label alignment results.}

Fig. 5. PSNR value comparison.

Fig. 6. F1 value comparison.}