3.1. Edge Extraction Module

To enhance edge feature learning, we incorporate a U-Net ^{[20- 22]} architecture into the edge extraction module. As shown in Fig. 3, this module aggregates multi-scale features from the backbone and compresses them into low-dimensional edge representations through a U-Net structure, where $C$ (experimentally set to 128) denotes the number of channels. The U-Net’s skip connections play a crucial role by bridging low-level encoder features with high-level decoder features, facilitating seamless cross-level information transfer and fusion.

Fig. 3. Edge extraction module based on UNet structure.

Assuming the size of the original image is $H \times W$, the size of the final output elliptic edge image can be obtained based on the scale factor $s_f$:

To obtain the correct ellipse parameters at the corresponding scale for the original 5-parameter label, all four parameters except the angle need to be divided by $s_f$. To create the label image, an initial single-channel grayscale image with all pixels set to 0 is essential. This image will serve as the background for the label.

The cv2.ellipse function can be used to create an elliptical mask with adaptively adjusted edge thickness. The thickness parameter $T$ is dynamically calculated using Eq. (2), where $w$ represents the long axis length of the ellipse and $\lfloor \rfloor$ represents round function.

Fig. 4 demonstrates the mask application in label generation. For clarity, a partial enlargement of the labeled image's edges is provided. The black squares denote single-pixel wide ellipse edges, which are defined as positive samples, while the gray regions represent masked areas excluded from training. This approach mitigates the impact of interfering samples and helps balance the ratio of positive to negative samples.

Fig. 4. Mask application in label generation: (a) Original elliptical edge annotations; (b) Processed labels with masking operation. The black edges represent positive samples, while gray regions indicate masked areas excluded from training.

During the training process, Focal Loss (FL) is used to optimize the learned edge images. The specific calculation process is shown in Eq. (3):

where $p_i$ represents the probability value of the $i$-th sample being predicted as a positive sample.

In Fig. 5, it can be observed that even in the early stages of training, the edge extraction module is already able to capture a significant portion of the elliptical edge images. However, at this stage, there is still considerable interference from other contours and noise, as seen in Fig. 5(b) where the overall contour of the bicycle frame can still be distinguished, and the elliptical edges in all three examples appear blurry. With continued optimization, as seen in Fig. 5(c), the edge extraction module eventually produces clear elliptical edges.

Fig. 5. Visualization of training results of the edge extraction module. (a) Original image. (b) Edge extraction results for training 2 epochs. (c) Edge extraction results for training 72 epochs. (d) Label image of the edge extraction module.

3.2. Rotated RoI Module for Ellipse Parameter Regression

In the edge extraction module, high-precision elliptical edge images have already been obtained. Therefore, the RoI parameter regression module utilizes convolutional layers to process the intermediate results of the edge extraction module, generating a new set of multi-scale features as its input. The specific generation process and structure are shown in Fig. 6, where $C$ (experimentally set to 128) denotes the number of channels.

Fig. 6. Multi-scale feature map generation process for RoI inputs using a UNet-based structure.

In order to achieve precise feature alignment in rotated objects before parameter regression, the RoI parameter regression module also employs the Rotated RoI Align, as can be seen in RoI Transformer ^[23]. In this paper, after processing by the RoI parameter regression module, the final size of the obtained RoI features is $128 \times 7 \times 7$. By doing so, this module can map each RoI region as a fixed-size feature vector and maintain the consistency of the number of channels.

The entire regression process is illustrated in Fig. 7. To perform parameter regression, we flatten these feature vectors, converting them into one-dimensional feature vectors. Then through several shared fully connected layers, two separate fully connected layers are used to regress accurate localization results and classification results. This design maps the RoI feature representation to the elliptical parameter space, enabling precise object localization.

Fig. 7. Regression process after RoI feature extraction.

Unlike typical two-stage object detectors, our network has already obtained better prediction results from the Oriented RepPoints detector. To reduce computational cost without compromising detection accuracy, the impact of different numbers of proposals (100, 200, 500, 1000) on detection performance was evaluated. Results indicate that using only 200 proposals achieves accuracy comparable to that of larger proposal sets. This demonstrates that the method can effectively cover target objects with fewer candidate regions, significantly reducing computational overhead while maintaining performance. In the case of Faster R-CNN, for example, it outputs 2000 candidate boxes from the Region Proposal Network (RPN) to the second stage of feature extraction, while our method uses only 10% of its computation. Due to the number of proposal boxes being drastically reduced and these boxes having inherently high classification confidence, the RoI parameter regression module should reduce the classification cost and focus more on the regression task.

Given that our target detection task essentially reduces to binary classification (distinguishing between foreground and background categories), the binary cross-entropy loss serves as the appropriate objective function for this classification component.

Its specific calculation procedure is shown in Eq. (4), where $p_i$ represents the label value of the $i$-th sample, with 1 indicating a positive sample and 0 indicating a negative sample. To optimize the localization results, as the localization results are regressed into a five-parameter representation of rotated bounding boxes, the SmoothL1 loss function is selected for optimization. It has a smoother gradient within a smaller error range, making it more resilient to outliers and exceptional values. The specific calculation process is as follows:

For the proposed edge-based auxiliary branch in this paper, three types of losses are involved: edge loss $L_{\text{edge}}$, RoI classification loss $L_{\text{cls}}$, and localization loss $L_{\text{box}}$. They can be combined by weighted summation, as shown in Eq. (6). Here, $\lambda_1$, $\lambda_2$, and $\lambda_3$ are the weight coefficients of the losses, used to balance the importance of different loss components. Unlike typical two-stage object detectors, since the detector Oriented RepPoints has already achieved high-quality detection results, the dependency on classification is significantly reduced in the RoI parameter regression module. Therefore, $\lambda_1$ and $\lambda_3$ are set to 1, while $\lambda_2$ is set to 0.2, emphasizing the optimization of edge loss and localization loss.

4.1. Dataset

This study focuses on high-precision ellipse detection in complex real-world scenarios. We utilize three publicly available datasets: the Prasad dataset ^[24], Random Images ^[25], and Smartphone Images ^[25], all containing JPG-format images captured in challenging environments with diverse edge complexities and varying image quality. To address the limited sample size of individual datasets, we combine them into a comprehensive collection comprising 1,127 images with 4,334 annotated ellipses. Following standard practice in computer vision research, we partition the dataset into training and test sets containing 983 and 244 images, respectively.

4.2. Implementation Details

All experiments were conducted using a single NVIDIA RTX 2080 GPU, following the baseline settings provided in the MMRotate toolbox. For the added U-Net branch, the loss weight was set to 1.0, and no additional hyperparameters were modified. The implementation used $512 \times 512$ input images throughout. Training converged in 2.8 hours (72 epochs), with final model metrics of 48.88 GFLOPs, 36.81M parameters, and 56.1 FPS inference speed. The traditional detection methods referenced in the paper were executed on a CPU, while Dong’s method ^[14] was implemented in accordance with the settings described in the original publication.

4.3. Evaluation metrics

In this paper, the performance of the proposed method will be evaluated by the effectiveness of detection, which will be realized through the following three metrics presented in Eq. (7). Precision measures the proportion of positive samples correctly identified by the detector, indicating the accuracy of the model. Recall measures the proportion of true positive samples detected, indicating coverage of the model. The F-measure provides a balanced assessment of precision and recall, serving as an indicator of the model's overall performance. True positives (TP), false positives (FP), and false negatives (FN) are utilized to evaluate the accuracy of the detector's predictions.

In addition, Average Precision (AP) is one of the most commonly used metrics to evaluate the performance of object detection models. It represents the area under the Precision-Recall (PR) curve. For example, in the case of AP50, if the Intersection over Union (IoU) between the predicted bounding box and ground-truth bounding box is greater than or equal to 0.5, the detection result is considered correct. In this paper, the performance of the proposed method will be comprehensively evaluated using both AP50 and AP80 metrics.

4.4. Ablation studies

An auxiliary branch comprising an edge extraction module and an RoI parameter regression module is introduced to facilitate high-precision detection of ellipses. Compared to the baseline method of Oriented RepPoints, the effectiveness of these two modules is validated through ablation experiments.

The first row in Table 1 shows the detection results of Oriented RepPoints as the baseline method. As shown in Table 1, the edge extraction module and ROI parameter regression module achieved 1.42% and 1.27% improvements in AP50 compared to the baseline, indicating that both modules have a positive impact on the detection performance of the model. In terms of AP80 performance, both modules showed more significant performance improvements, especially the ROI parameter regression module, which was 4.07% higher than the baseline. When both modules were added, AP80 improved by 6.53% compared to the baseline. The ablation experiment demonstrated that our proposed auxiliary branch significantly improved detection performance at high IoU thresholds.

4.5. Comparison with other methods

This paper compares the training results of the proposed method with those of the baseline and three state-of-the-art ellipse detection algorithms. Among them, Arc-Support and AAMED are traditional methods, widely recognized in the literature as the most effective conventional ellipse detectors with publicly available source code. Additionally, Dong’s method ^[14], a deep learning-based approach, is included in the comparison and is trained for 72 epochs, in alignment with our training protocol. To evaluate detection performance under high localization accuracy requirements, comparative experiments were conducted using overlap ratio thresholds of both 0.5 and 0.8. The performance evaluation focuses on Precision(P), Recall(R), and F-measure, and the results are presented in Table 2.

When localization requirements are not strict, deep learning-based methods often achieve better detection results due to their ability to process complex scenarios compared to traditional algorithms. As can be seen from Table 2, when the overlap ratio is 0.5, three deep learning-based ellipse detection methods show superior detection performance. Our proposed method excels in all three metrics, achieving a slightly higher recall compared to the baseline with a significant 7.61% improvement in precision.

When the overlap ratio is set to 0.8, the traditional methods Arc-Support and AAMED maintain relatively high precision and recall, whereas the baseline and Dong’s deep learning-based method ^[14] exhibit noticeably inferior performance. The proposed method effectively addresses the limitations of deep learning-based approaches in achieving high-accuracy ellipse detection and demonstrates superior performance across all three evaluation metrics. Specifically, it surpasses the AAMED method by 5.71% in precision and 1.43% in recall. The detection results of all compared methods are visualized and analyzed in Fig. 8.

Fig. 8. Visualization of detection results for each comparison method. (a) Arc-Support [8]. (b) AAMED [9]. (c) Dong [14]. (d) Baseline [16]. (e) Proposed. (f) Ground truth.

As shown in Fig. 8(b), the AAMED algorithm can accurately detect and locate ellipses in most cases, but there are also some false positives. In contrast, the false positives and inaccurate localization of Arc-Support in Fig. 8(a) are more obvious. As for deep learning-based methods, although Dong's method ^[14] (Fig. 8(c)) and the Baseline method (Fig. 8(d)) capture most of the ellipses, there are still missing detections and inaccurate shapes compared to ground truth. With the auxiliary branch, our proposed method (Fig. 8(e)) exhibits the best performance in this experiment, not only with accurate localization, but also with almost no false positives.