2.1 Secure Image Display

According to a study ^[13], one image to be protected by the SID should produce PN $\textit{grille images}$ (GNs) with non-overlapping virtual fences. The generated GNs are played back sequentially and repeatedly so that the image is securely displayed on a display screen. PN is a playback period that reduces design complexity to generate GNs. The virtual fence should be generated to conform to the following four rules. First, the PN value should be greater than or equal to 2. Second, during the PN period, the union of the image area uncovered by fences should be the whole image. Third, the current fence area should not be overlapped with the previous and next fence areas in order to use the afterimage effect. Fourth, if the combined area of the fence and the space between the fence is multiplied by an integer M, it should become the entire image area. The last rule makes the fence design much easier.

The rationale of using a virtual fence is the afterimage effect, which has been mathematically modelled ^[14]. Even though the model does not precisely describe the complex human visual system, the factors that influence the quality of the image perceived by the human eye are at least revealed.

2.2 Object Detection

The object detection technique is necessary under the assumption that the value of copyright protection in an image is focused on objects rather than the background. This assumption is reasonable because most animation characters are not treated as background. Object detection has to solve two main problems: object classification and localization. Some region-based CNN (R-CNN) algorithms ^[20-22] use two stages: localization first and then classification. These two-stage detectors have high classification accuracy, but the region proposal consumes much computational power, mainly due to the complexity of the selective search method.

Single-stage detectors ^[18,19] (also known as the YOLO family) try to simulate a human visual system that performs both classification and localization at the same time. R-CNN methods generally detect an object more accurately, but they tend to have slower speed and higher background errors than YOLO algorithms. In this study, object detection was performed using YOLO because it is important to detect the content part, which is the rest of the background, rather than an exact object classification. Also, the fast operation of YOLO is suitable for a real-time application. The output of a YOLO network is rectangular bounding boxes with a label. If only one object is detected, applying a virtual fence method would not be difficult. However, there may exist multiple objects and bounding boxes frequently overlapping each other. Virtual fence generation rules ^[12] are highly unlikely to be satisfied due to this situation. Therefore, we propose a novel concept of protection zone, which can be determined based on object bounding boxes.

Fig. 1. Block diagram of overall proposed object-wise secure image display method.

3.1 Protection Zone Localization

The simple protection idea is that virtual fences are applied independently to each bounding box detected by an object detection method. However, objects in an image are mostly not separated from each other, so it is very likely that the bounding boxes surrounding the objects overlap each other. If virtual fences are separately applied to individual bounding boxes, the overlapped area cannot satisfy the virtual fence generation rules summarized in Section 2.1. This causes severe distortion of the perceived image quality inside the overlapped area because some of the overlapped area can be too frequently occluded by fences. The objective of this study is not to detect objects in an image, but to protect a valuable part of an image from copyright infringement, so we need to define a rectangular protection zone that is bounded based on bounding boxes.

To create a protection zone, the overlapped bounding boxes should be repeatedly merged until there are no overlapping regions. Two overlapped bounding boxes are merged to make a temporary protection zone in the process of finding the final protection zones. The temporary zone should contain all merged bounding box areas and should be rectangular in shape. Therefore, some background area can be included in that temporary protection zone. This process can cause a temporary protection zone to overlap with bounding boxes that did not originally overlap, which requires another merging between them. Such examples are shown in Figs. 3(a) and (b) and will be described in more detail later. The localization process of final protection zones is done when there are no more intersections between bounding boxes and temporary protection zones.

A bounding box $b_{n}$ can be represented with two coordinates, $(b_{n}x_{1},b_{n}y_{1})$ and $(b_{n}x_{2},b_{n}y_{2})$, which are the top-left and bottom-right points of the box, respectively. Similarly, the temporary protection zone $p_{n}$ can be represented with the top-left point $(p_{n}x_{1},p_{n}y_{1})$ and the bottom-right point $(p_{n}x_{2},p_{n}y_{2}).$ The final protection zones are represented by $p_{fn}.$

The localization of a protection zone can be divided into three steps as follows. First, we check whether there are two overlapping bounding boxes. If any, two bounding boxes $b_{1}$ and $b_{2}$ should be merged so that the temporary protection zone $p_{1}$ is created and is represented using Eq. (1). Second, after creating $p_{1}$, which contains $b_{1}$ and $b_{2}$, $b_{2}$ is deleted in order to mark it as "already inclusive." Third, the temporary protection zone $p_{1}$should be checked for whether it is overlapped with $b_{3}-b_{N}$. If overlapped, $p_{1}$ continues to merge overlapping bounding boxes. Otherwise, the temporary protection zone becomes the final protection zone. A single protection zone is created after this process. If there are still overlapping bounding boxes, another protected area can be generated.

The detailed algorithm is shown in Fig. 2. We define a list $\alpha $ of $b_{n}$ detected in a single image. The list is in the form of a Python dictionary. The list $\alpha $ contains the labels $b_{n}$ and the corresponding coordinates. A (key, value) pair of the label and its corresponding coordinate is represented by $O_{n}$.

The overall structure goes through the process of updating $O_{n}$ to the temporary protection zone until there is no more overlap. Localization is the merge process between each $O_{n}$ and is described as follows. The first $b_{1}$ is copied to $p_{1}$. $p_{1}$ is created in a list named $\beta $ to be used as a temporary protection zone. This is done because when nothing overlaps until the end of $\alpha $, $b_{1}$ becomes a temporary protection zone $p_{1}$ by itself without any further operation. $p_{1}$ sequentially searches for whether or not it overlaps from $O_{2}$ to $O_{N}$ inside $\alpha $. At this time, self-comparison with one's own dictionary is excluded. If $p_{1}$ overlaps $b_{n}$, we update the coordinates of $p_{1}$ through Eq. (1). $b_{n}$ is removed from $\alpha $ to avoid duplication in the next search in the loop. This means that as the localization process occurs, the size of $\alpha $ decreases in most cases. $p_{1}$ continues the search process from the next element $b_{n+1}$. If $p_{1}$ goes through the loop to the end of $\alpha $, the previous search process repeats recursively starting with the next unremoved element on the list, $b_{2}$. When the recursive function is finished, the $\beta $ that stored $p_{n}$ is updated to $\alpha $. Now, $\alpha $ consists of $p_{1}-$$p_{n}$. If there are still overlapping elements in $\alpha $, the localization function is repeated again. Finally, if there are no more overlapping boxes, $\alpha $ consisting of $p_{f1}-$$p_{fn}$ becomes the output.

An example of the protection zone localization algorithm is illustrated in Fig. 3. Fig. 3(a) shows six bounding boxes $b_{1}-b_{6}$ detected by the YOLO network. Initially, $b_{2}$ and $b_{5}$ overlap, and so do $b_{3}$ and $b_{6}$, so they need to be merged later on. The subscripts are numbered in the order the bounding box that is stored in the list $\alpha $. $p_{1}$ is a temporary protection zone where nothing one overlaps with others during the search process.

A protection zone $p_{2}$ begins to be created in Fig. 3(c) by merging $b_{2}$ and $b_{5}$ according to the algorithm in Fig. 2. In this case, $p_{2}$ merges because it shares some area with $p_{1}$, which originally did not overlap. In Fig. 3(d), $b_{3}$ finds $b_{6}$ overlapping, and $p_{3}$ is generated by merging with it. $b_{4}$ becomes the temporary protection zone $p_{4}$ by itself without any further operation, as illustrated in Fig. 3(e). $p_{2}$ extends to another temporary protection zone, including the object of $p_{1}$. In this case, $p_{f1}$ becomes the final protection zone since there are no further overlapping boxes or protection zones. Also, $p_{3}$ is the final protection zone $p_{f2}$. Similarly, $p_{4}$ becomes the final protection zone $p_{f3}$ by itself without any further operation, as illustrated in Fig. 3(f).

The proposed protection zone localization algorithm was applied to digital animation resources, and some examples are shown in Fig. 4. We discarded the labeling information from the YOLO output and used only the bounding box information. The complex and various boxes that existed in the first place were neatly arranged to create an area worth protection, which shows that the protection zone localization algorithm is effective. There are some problems with the final protection zones, so we need protection zone adjustment, which will be discussed in the next section.

Fig. 2. Protection Zone Localization Algorithm.

3.2 Protection Zone Adjustment

After localization of the protection zones, virtual fences must be generated inside them to create a protected image. The virtual fence generation rules summarized in Section 2.1 can also be applied to protection zones with some additional rules. Therefore, we can create virtual fences that move at the same playback rate and exposure rate in every protection zone. The need for additional rules arises from the difference between the entire image protection method and the adaptive image protection method proposed in this study.

We found three additional problems with using virtual fences for protection zones in all situations. First, there is a small protection zone problem, as shown in Fig. 4(a). There is a possibility that the size of the detected object is very small. This is not a problem if the small bounding boxes are merged in the process of protection zone localization. Otherwise, applying a virtual fence to these small protection zones increases image distortion with little to no protection benefit. Since the purpose of this study is to protect valuable parts of the image, we remove protection zones that do not reach the appropriate size. The threshold size is set to 10\% of the image size.

Second, there is a variable protection area problem, as shown in Figs. 4(a)-(c). While the virtual fence generation rules are applied to fixed size images, the protection zone size is variable. The image size used in another study ^[13] was deliberately predetermined so that the entire image area is equal to the sum of the fence area and the area between the fences multiplied by the integer M=5. As a result, the virtual fence generation rule is hard to follow in protection zones of varying sizes. Of course, the solution is to increase the size of the protection zone so that the fixed exposure rate and PN can be physically applicable conforming to the fence generation rules. The size is also increased so that the valuable area is not exposed by the modification. However, if one protected zone meets another during the change, it must be reduced in size as an exception. One thing we need to remember is that we do not have to change the height of the protection zone because the fence generation rules do not control the fence height

Third, there is a multiple-protection-zone problem, as shown in Fig. 4(c). The way to cope with this problem is to ensure the same image quality in all protection zones. The image quality depends on the playback rate and exposure rate. Since the playback rate affects every protection zone, we do not have to do anything regarding this. However, the size of the protection zone should be adjusted to keep the same exposure rate in all protection zones. The largest protection zone is selected, and its size is changed according to the solution of the second problem. For other protection zones, the fences are applied using the ratio of the largest zone width to the width of the other zones. The anchor point of the size change is the upper left coordinate of the protection zone, so the direction of increasing or decreasing smaller protection zones is on the right side. If it reaches the end of the image, the protection zone is virtually expanded. The virtually expanded area may have virtual fences, but they are not actually visible on the screen. Some results are shown in Fig. 5.

Fig. 3. An example of protection zone localization.

Fig. 4. Three final protection zone examples.

Fig. 5. Sample from Table 1 at 0.573 seconds.