AFOD: Two-stage object detection based on anchor-free remote sensing photos

3.1 Algorithm framework structure

In this article, the algorithm’s backbone network utilizes ResNet-50 and incorporates augment feature pyramid networks (AugFPNs) to merge the characteristics from each feature map. Given the prevalence of small objects in remote sensing photos and the complexity of the images, the algorithm in question employs AugFPN to enhance the precision of detecting small-sized targets. Meanwhile, in order to better detect the tilted and relatively large objects in length and width, this article adds an additional parameter θ ∈ [0, π/2], which refers to the angle between the short edge of the enclosing frame and the horizontal direction, in addition to the anchor-free approach. The initial feature map identifies the object’s location in the image by applying the SAM. Subsequently, the anchor-free rotating frame object detection module determines the confidence level and the bounding frame. Utilizing the coordinates of the bounding frame, the relevant segment of the feature map is chosen, and the RoIAlign module produces two feature maps of the region of interest, sized 7 × 7 and 3 × 11. These maps are further enriched by the CAM. The accurate detection results are ultimately derived using the R-CNN detection head module. The algorithm’s structure is illustrated in Figure 1.

3.2 Augment feature pyramid

The AugFPN proposes consistent supervision, which applies the same supervisory signal to features at different scales before feature integration to attenuate the differences between their semantic information. First, a feature pyramid is constructed based on multiple feature layers {C2, C3, C4, C5} of the backbone network, and then multiple RoI are generated at the feature pyramid layers {P2, P3, P4, P5} using the RPN, and map all the RoI to each feature layer, and get the features of the {M2, M3, M4, M5} RoI in each layer by RoIAlign, and finally add the branches used for prediction to get the auxiliary loss, as shown in Figure 2.

Figure 2

AugFPN structure diagram.

The network structure allows sharing the parameters on all layers into the output and fusing features from different layers to obtain more semantic information. Since the auxiliary loss and the predicted loss have different effects on the model, an additional weight is needed to balance them, as follows:

where L _cls,M and L _loc,M denote the losses on the feature map {M2, M3, M4, M5}, L _cls,P and L _loc,P represent the losses on the feature pyramid {P2, P3, P4, P5}. The variables n _M and n _P , respectively, refer to the total number of predictions at the Mth level of the feature map at the Pth level of the feature pyramid. Furthermore, p _M and d _M are the prediction results of the feature map, while p and d represent the feature prediction results of the pyramid. The terms t ^* and b ^* indicate the confidence and regression values of the true box, respectively. λ serves as the parameter for adjusting the proportion of auxiliary and original losses, whereas β is the parameter used for regulating the proportion of categorical and regression losses. Additionally, [t ^* ] is defined as follows:

3.3 AFOD box

The method in this article uses anchor-free object detection to produce rough orientation detection frames and confidence levels. The rough orientation detection box includes distance and angle, which are denoted by l, r, t, b, and θ, respectively, and the confidence level is denoted by c. Let B _g = (x _g,y _g,w _g,h _g,θ _g) be the ground truth box of an object, where B _g ∈ ℜ ⁵ × C, (x _g, y _g) denotes the center coordinates of the object, w _g and h _g represent the object’s width and height, correspondingly. θ _g represents the counterclockwise rotation angle of the actual bounding box, while C stands for the category number. In the classification branch, for each position (x, y), if it lies in the central region B _δ = (x _g, y _g, δw _g, δh _g, θ _g) of the real frame, and δ denotes the centrality, the position is labeled with a positive sample. The disparity between each site and the actual position is

If both |∆x| < δw _g/2 and |∆y| < δh _g/2 are satisfied, the position is a positive sample, as shown in Figure 3.

Figure 3

AFOD box.

In the regression branch, each positive sample is represented by d = (l,r,t,b,θ), indicating the distances from (x, y) to the left, right, upper, and lower bounding boxes, alongside the angle between the lower edge and the positive X-axis, where the values of l, r, t, b are calculated as follows:

3.4 RoIAlign

The implementation of RoIAlign in this article is shown in Figure 4. RoIAlign is based on the bilinear interpolation method, which uses the four nearest pixels on the feature map to obtain the pixel values, which are usually floating-point coordinates, so the interpolation method is used to obtain the pixel values. The output is the maximum of the four sampling points. In order to adapt to objects with large length and width, the regions of interest of 7 × 7 and 3 × 11 sizes are selected as the output. Ultimately, the CAM is leveraged to accentuate the distinctive foreground attributes while subduing the background elements.

Figure 4

RoIAlign.

3.5 Faster R-CNN head

Utilizing the faster R-CNN detection head, we derive the ultimate classification outcomes and the localization frame coordinates. Within this component, the input feature map is transformed into a feature vector through two fully connected (FC) layers, later divided into a classification segment and a regression segment. The classification segment employs a single fully connected layer to forecast the classification score; in the regression segment, another parameter θ is added for the regression of the angle, i.e., the bounding box of the regression is denoted as (l, r, t, b, θ), where θ ∈ [0, π/2], in order to obtain the rotated detection frame, the final regression result is obtained by one fully connected layer. More accurate classification scores and directed detection boxes can be predicted using this module.

3.6 Loss function

The loss function in this article contains two components, the function L _R-CNN of the augmented feature pyramid and the function L _head of the detection head, where L _R-CNN is shown in equation (1) and L _head is as follows:

where L _cls and L _loc denote score loss and coordinate loss, respectively, used in this article are softmax and smooth-L1, respectively. The score value and coordinate value of the authentic frame are denoted as z* and d*, respectively. n denotes the t total sample count across the feature pyramid, and [z*] is defined as follows:

The total loss function of the algorithm in this article is

4.1 Evaluation metrics

The model presented in this article utilizes the mean average precision (mAP) as its evaluation metric. In target detection, there are usually three types of detection results: correct detection boxes (TP), false detection boxes (FP), and missed detection boxes (FN). For these three cases, the accuracy (precision) and recall (recall) test criteria are used to verify the capability of the target detection algorithm. They are calculated as follows:

The mAP is the result obtained by detecting the average accuracy of all categories and then taking the average. In the detection process, the accuracy and recall of each prediction frame are calculated, and the point (P, R) is obtained. Each time an R value is acquired, it serves as the threshold for the subsequent calculations. When the R value surpasses this threshold, the maximum accuracy for the corresponding scenario is determined. By computing the average of these accuracies, the AP is derived, and subsequently, the mAP is computed from the collective average of all classified APs.

4.2 Datasets

The experimental data presented in this study primarily rely on two publicly accessible datasets for aerial remote sensing images: the DOTA datasets [34] and the HRSC2016 datasets [39]. The DOTA datasets consist of 2,806 aerial images with varying scene sizes, ranging from 800 × 800 to 4,000 × 4,000. These images encompass 188,282 instances distributed across 15 categories. The labeling technique involves a quadrilateral delineated by four points, allowing horizontal bracketing box and oriented bracketing box (OBB) detection tasks. Among the images, 1,411 are assigned to the training set, 458 to the validation set, and 937 to the test set. The classes within the DOTA datasets include plane (PL), bridge (BR), port (HA), athletic field (GTF), small vehicle (SV), large vehicle (LV), roundabout (RA), swimming pool (SP), ship (SH), soccer field (SBF), tennis court (TC), basketball court (BC), oil storage tank (ST), baseball field (BD), and helicopter (HC). On the other hand, the HRSC2016 datasets comprise 1,061 images and 2,976 instances captured from Google Earth, annotated in an oriented bracketing box (OBB) format. The dataset includes 436 images and 1,207 instances in the training set, 181 images and 541 instances in the validation set, and 444 images and 1,228 instances in the test set, constituting a comprehensive high-resolution detection dataset. Notably, for the purposes of this study, all ships within the datasets are grouped under a singular “ship” category during the training and testing phases.

4.3 Experimental results

The validation results for our approach to the DOTA datasets are depicted in Figure 5. Table 1 displays a comparison between the accuracy of our method and other models on the same datasets, where bold numbers indicate the currently optimal performance. To ensure the impact of the backbone network does not affect the results, all the compared models in this study utilize Resnet-50 or Resnet-101 and FPN as the feature extraction network, thus demonstrating the efficacy of the model structure. As can be seen from Table 1, the mAP of the method in this article reaches 77.18%, and the AP of this article is improved by about 2% compared with other methods in the categories of aircraft, large vehicles, and swimming pools, and about 3% in the categories of athletic fields and small vehicles, and even about 4% in the category of roundabout intersection. Compared to the top-performing one-stage model, AFOD further improves the mAP by 3.06%. First, the AFOD model utilizes an anchor-free approach, which, in contrast to traditional anchor-based methods, may suffer from limitations imposed by anchor settings in certain scenarios, leading to a certain degree of precision loss. Second, the AFOD model incorporates SAMs and CAMs, which help the model focus more on crucial features, extract and strengthen useful information, thereby contributing to improved accuracy in target detection. Finally, the AFOD model is based on a rotation framework specifically designed for directional targets. This enables the model to better adapt to the presence of rotating targets in the DOTA dataset, accurately capturing the orientation information of the targets. In contrast to the two-stage method, the gliding vertex technique aims to minimize the sequential label point confusion by smoothly moving vertices along the sides of the horizontal bounding box to depict objects with multiple orientations. Additionally, it incorporates an obliquity factor dependent on the area ratio to determine whether to opt for horizontal or oriented detection. However, the gliding vertex approach could face constraints when handling rotated objects. In contrast, the AFOD model, by removing the anchor setting and introducing additional angle parameters, can more accurately capture targets in the rotating frame, effectively addressing the limitations of traditional horizontal detection frameworks in multi-oriented object detection.

Figure 5

DOTA dataset visualization results.

The validation results of our method on the HRSC2016 datasets are depicted in Figure 6. Table 2 presents a comparison between the accuracy of our approach in this article and that of alternative models on the same datasets, where bold numbers indicate the currently optimal performance. On the HRSC2016 dataset, although the RoI transformer has introduced a novel RoI transformation module that can optimize localization and feature extraction, it may not comprehensively address the challenges of object detection in a rotating framework. In contrast, the AFOD model employs an anchor-free approach based on the rotating frame in its proposed algorithm, combined with spatial attention and CAMs. Consequently, it can more accurately locate and identify targets in high-resolution aerial photographs, achieving an mAP of 88.15%.

Figure 6

HRSC2016 datasets visualization results.