Line segment using displacement prior

2.1 Deep line segment detection

M-LSD [23] uses tri-points to predict line segments, producing superior results in model parameters, network complexity, efficiency, and precision. Therefore, this article aims to dive deeper into line segment detection using M-LSD. The backbone of M-LSD is an encoder–decoder structure, the MobileNetV2 [30] network is used as the encoder structure of the backbone network, and the decoder is designed as a top-down architecture.

As shown in Figure 1, M-LSD outputs 16 feature maps, including center maps, displacement maps, angle maps, length maps, line map, and endpoint map. Each feature map corresponds to a labeled map of the same resolution. To address the limited detection performance of the model for long line segments caused by an insufficient network receptive field, M-LSD uses the segments of line (SoL) augmentation. The specific approach involves splitting a long line segment into several shorter line segments that are treated as a separate entity for prediction. The SoL maps do not play a direct role in the final line segment prediction; instead, they offer auxiliary information to aid in the prediction of line segments.

Figure 1

Overall architecture of M-LSD-tiny.

As shown in Figure 2, the center map is used to predict the centers of line segments, while the displacement maps are used to predict the displacements from the centers to the endpoints of the line segments:

where d s and d e denote the displacements from the center of the segment to the two endpoints, respectively.

Figure 2

Tri-points representation of line segments.

The centroid-based representation avoids the significant computation required by direct endpoint matching and greatly enhances the detection efficiency of line segments. Compared with representing line segments based on length and angle, the tri-points representation of line segments can make the model more sensitive to angles, particularly for long line segments. In this article, the tri-points representation will be utilized to predict line segments.

2.2 Deformable convolution

In visual tasks, the learning process essentially involves adjusting the model parameters to adapt to the size, shape, and posture of the object. Vanilla convolutional kernels sample the feature map at a fixed position and apply the same weight to each position to process the feature. However, they lack the processing ability for objects with irregular shapes, making it difficult to accurately capture the shape and position of the object. Since different positions in the feature map may correspond to objects of various sizes and shapes, convolution is necessary to adaptively adjust the sampling position and receptive field size based on different objects for visual tasks that demand high precision. In 2017, Dai et al. [25] proposed deformable convolution to enhance the ability to express deformable characteristics in processing objects. The spatial deformation was introduced to allow the convolution kernel to generate positional deviations when sampling from the input feature maps.

In 2D convolution, sampling occurs at a fixed position R , which determines the receptive field of the convolution kernel. For a convolution kernel of size 3 × 3 , R = { ( − 1 , − 1 ) , ( − 1 , 0 ) , … , ( 0 , 1 ) , ( 1 , 1 ) } , the corresponding output at p 0 in the output feature map y is as follows:

where x represents the input feature map, p n is a vector in R , and w is the weight of the convolution kernel.

As shown in Figure 3, deformable convolution introduces a set of offsets { Δ p n ∣ n = 1 , … , N } , where N = ∣ R ∣ , obtained through network learning, to the fixed sampling points of ordinary 2D convolution. This allows the receptive field of the convolution kernel to be adaptively tuned to the shape of the object, rather than capturing features in a fixed shape. The output feature of deformable convolution can be expressed as follows:

Deformable convolution adjusts the sampling position of the convolution based on the object’s shape. This adjustment ensures the validity and accuracy of the features extracted by the convolution kernel, allowing the network to model the object more precisely and enhancing its robustness to shape variations.

Figure 3

Illustration of 3 × 3 deformable convolution.

2.3 Datasets

In this article, the model will be trained and tested using the Wireframe dataset [15] and the YorkUrban dataset [6], 5,000 images from the Wireframe dataset, while the test set will include 462 images from the Wireframe dataset and 102 images from YorkUrban. In the training process, we perform the following dataset augmentations: (1) unchanged; (2) horizontal flip, vertical flip, or horizontal and vertical direction flip simultaneously; (3) rotate 90 degrees clockwise or counterclockwise; (4) randomly crop the image, and then resize it to a resolution of 512 × 512 .

3.1 Overall network architecture

As shown in Figure 4, our proposed network is one-stage and consists of a backbone and a prediction head. The backbone takes an image of size 512 × 512 × 3 as input and outputs the shared features with a size of 256 × 256 × 64 . The prediction head consists of three branches. The first branch consists of a 3 × 3 deformable convolution, a 3 × 3 vanilla convolution, and a 1 × 1 vanilla convolution connected sequentially. It takes shared features as inputs and produces displacement maps as outputs. The second branch consists of a 3 × 3 deformable convolution using displacement prior (Displace-Deform Conv), a 3 × 3 vanilla convolution kernel, and a 1 × 1 vanilla convolution connected sequentially. It takes as inputs the shared features and the output displacements of the first branch, and produces the center map as output. The third branch consists of a sequence of 3 × 3 deformable convolution, 3 × 3 vanilla convolution, and 1 × 1 vanilla convolution. These components are connected sequentially, sharing inputs and feature maps. The branch outputs a junction map and a line map, which only offer auxiliary information for the training process and are not directly involved in the final line segment prediction.

Figure 4

Overall architecture of D-LSD.

In the final feature maps, displacement maps predict the displacements of line segments, the center map predicts the centers of line segments, the junction map and line map predict junctions, and the pixel-wise map predicts the line segments’ details. During training, displacements are regressed using smooth L1 loss, while centers, junctions, and pixels are trained using focal loss.

3.2 Backbone network

HRNet is a deep learning network architecture used for human pose estimation. It has also been employed as a backbone network for line segment detection, demonstrating promising results. The decoder of UNet3+ fully integrates high-level semantic features and low-level semantic features in the full-size feature map through full-size skip connections, allowing information to be more comprehensively transmitted within the network and effectively enhancing model performance. As shown in Figure 5, our backbone network features an encoder–decoder structure. To enhance the network’s ability to extract line segment features, we utilize HRNet as the encoder of the backbone network. In order to fully integrate features at different levels, capture detailed information at various scales, and enhance the network’s capability to represent features such as fine-grained structures and edges in the image, the decoder of UNet3+ is selected as the decoder of the backbone network.

Figure 5

Overall architecture of the backbone network.

The decoder network takes an image of size 512 × 512 × 3 as input and integrates features of various resolutions through parallel sub-networks. It ultimately produces five sets of feature maps with resolutions of 256 × 256 , 128 × 128 , 64 × 64 , 32 × 32 , and 16 × 16 . The decoder takes the output of the encoder as input and fully integrates the high-level semantic features and low-level semantic features in the full-size feature maps to enhance the network’s ability to retain details. When fusing feature maps with different resolutions to reduce the number of network parameters, the number of channels is standardized to match the number of channels in feature maps with a resolution of 128 × 128 . Finally, block B of M-LSD is utilized in the output layer of the backbone network to conduct feature fusion on the output feature maps of the decoder layer in order to produce shared features.

3.3 Deformable convolution using displacement prior

In this section, the predicted displacements are used as prior information to guide the sampling position of the deformable convolution in the center branch. As shown in Figure 6, the offsets of the deformable convolution for the center branch do not need to be obtained by convolution; instead, they are entirely derived from the prediction results of the displacement maps.

Figure 6

Center branch using displacement prior.

Let the shared feature be f , and the center be p 0 ; the offset from the vanilla convolutional sampling point to the center point is p 0 , p n ∈ { ( − 1 , − 1 ) , ( − 1 , 0 ) , … , ( 0 , 1 ) , ( 1 , 1 ) } . The corresponding predicted displacements for p 0 are p s and p e . Therefore, the offset from the sampling point to the center point of the convolution is as follows:

The offset of the deformable convolution can be expressed as follows:

The value at p 0 in the output feature map f ′ can be expressed as follows:

The offsets of the deformable convolution using the displacements are illustrated in Figure 7, where all the sampling points are distributed between the center and the endpoints.

Figure 7

Deformable convolution using prior displacements.

3.4 Dynamic label assignment for the center of line segment

In M-LSD, the centers of line segments are predicted by working with a feature map of size 256 × 256 . As shown in Figure 8 (a), all pixels within a 3 × 3 range around the centers are labeled as positive samples represented by “1,” and the remaining pixels are considered negative samples represented by “0,” respectively. In this article, we propose a dynamic center label assignment strategy. As shown in Figure 8 (b), the pixels are divided into positive samples, ignore samples, candidate negative samples, and negative samples according to the distance to the centers. For a pixel in the feature map, the distance between it and the centers of all line segments in the label is calculated as follows:

where x p and y p denote the x -axis and y -axis coordinates of pixel p , x l c , and y l c denote the x -axis and y -axis coordinates of the center point of line segment l , and G denotes the set of labeled line segments corresponding to the feature map. When the condition d c ( p , l c ) ≤ 3 is satisfied, p is labeled as a positive sample; otherwise, it is labeled as a negative sample.

Figure 8

Center label of line segment: (a) is the static center label assignment strategy used by M-LSD: “1” represents a positive sample and “0” represents a negative sample, respectively; (b) is our dynamic center label assignment strategy: “X” means ignore sample, “?” represents the candidate negative sample based on the its displacement prediction: (a) center label in M-LSD and (b) dynamic center label.

The displacement maps predict the vectors from the centers to the endpoints, and the final predicted line segment can be obtained from the center map and the displacement maps:

where x l s and y l s denote the x -axis and y -axis coordinates of the start point of line segment l , x l e and y l e denote the x -axis and y -axis coordinates of the end point of line segment l , d s ( x l c , y l c ) , and d e ( x l c , y l c ) denote the displacements from the center of line segment l to the start points and end points, respectively.

The line segment is predicted by both the center and the displacement. During the training process, the proposed center label assignment strategy dynamically assigns whether a candidate center negative sample is classified as a negative sample or an ignore sample according to its segmentation results. For the candidate negative sample, based on its displacement, if the line segment result is accurate, it should be classified as an ignored sample. The accuracy of the line segment is calculated based on the distance between the predicted endpoints and the label endpoints:

where l ˆ denotes the predicted line segment and l denotes its label line segment. d ( l ˆ s , l s ) and d ( l ˆ e , l e ) denote the distance between two predicted endpoints and the label endpoints, respectively, which can be calculated by

If d ( l ˆ , l ) ≤ γ , γ is the distance threshold between line segments, then p is not considered a negative sample. The specific process is shown in Algorithm 1.

Furthermore, the other pixels surrounding the centers no longer share the displacements of the centers. Instead, they are computed separately for each of their respective displacements to the endpoints of the line segments. This enhancement significantly improves the accuracy of the displacements.

The dynamic label assignment strategy enables the network to concentrate more on learning to classify the center and the background without allocating excessive attention to the regions surrounding the centers, which have less impact on the results. This approach allows the network to demonstrate improved performance.

4.1 Ablation study

In order to verify the effectiveness of the dynamic label assignment strategy and the backbone network proposed in Section 3, the following ablation experiments will be conducted using M-LSD-tiny as a benchmark. All experiments will be carried out on NVIDIA GTX 3080Ti GPUs using the PyTorch framework. The model parameter settings and optimization will be consistent with those of M-LSD-tiny.

As shown in Table 1, we use M-LSD-tiny as baseline for the ablation experiments. Replacing the label assignment strategy in baseline with our dynamic label assignment strategy, the F H , s A P 5 , and s A P 10 metrics improve by 0.9, 1, and 1.2% in Wireframe dataset and by 2.7, 3.2, and 2.0% in YorkUrban dataset. Replacing the encoder and decoder of the backbone network with HRNet32 and UNet3+, respectively, the F H , s A P 5 , and s A P 10 metrics improve by 3.2, 18.0, and 14.8% in Wireframe dataset and by 4.2, 14.9, and 11.2% in YorkUrban dataset. Then, simultaneously using the dynamic label assignment strategy and backbone network proposed in this article, the F H , s A P 5 , and s A P 10 metrics improve by 3.6, 18.4, and 15.3% in Wireframe dataset, and by 4.3, 16.7, and 12.8% in YorkUrban dataset.

We use M-LSD-tiny as baseline, Dis-Deform denotes replacing Block C with the prediction head shown in Section 3.3, and Deform denotes changing the deformable convolution using displacement prior in the prediction head of Section 3.3 to normal deformable convolution. As shown in Table 2, using the prediction head proposed in this article, the F H , s A P 5 , and s A P 10 metrics of the detection results are improved by 2.1, 13.6, and 10.9% in Wireframe dataset, and by 3.7, 13.6, and 11.2% in YorkUrban dataset. Compared to using normal deformable convolution, deformable convolution using displacement prior has 0.2, 3.5, and 2.9% higher for F H , s A P 5 , and s A P 10 metrics in Wireframe dataset, and 1.4, 5.0, and 4.9% higher for F H , s A P 5 , and s A P 10 metrics in the York dataset, respectively.

Then, simultaneously using the dynamic label assignment strategy, prediction head and backbone network proposed in this article, the F H , s A P 5 , and s A P 10 metrics on the wireframe dataset are 81.0, 64.4, and 68.7, respectively, and on the York dataset, the F H , s A P 5 , and s A P 10 metrics are 65.2, 27.8, and 30.4, respectively.

In addition, two non-maximal suppression methods, SoftNMS and StructNMS, proposed by F-Clip [21] are applied to the post-processing algorithm of our proposed algorithm, and the results is shown in Table 3. Eventually, the proposed model achieves 81.2, 65.5, and 69.5 for F H , s A P 5 , and s A P 10 metrics in Wireframe dataset, and 65.3, 29.5, and 32.2 for F H , s A P 5 , and s A P 10 metrics in YorkUrban dataset, respectively.

4.2 Comparison with other methods

The results of comparing our proposed line segment detection algorithm with other top-performing line segment detection algorithms such as HAWP, L-CNN, and F-Clip on Wireframe dataset and YorkUrban dataset are shown in Table 4. On the Wireframe dataset, F H , s A P 5 , and s A P 10 metrics are 2.0, 0.8, and 0.3% higher than the current top-performing ELSD algorithm, respectively, and on the YorkUrban dataset, F H , s A P 5 , and s A P 10 metrics are 3.5, 4.5, and 8.0% higher than the current top-performing F-Clip algorithm, respectively. The performance lead of our proposed algorithm over other algorithms is higher on the YorkUrban dataset than on the Wireframe dataset, which proves that the model is also better in terms of robustness and generalization.

Among the existing methods for line segment detection, two-stage methods demonstrate better performance. The method described in this article utilizes displacements as prior information to guide the sampling process of deformable convolutions for the center branch and the label assignment of centers. Finally, our method achieves better detection performance with a one-stage structure than the existing two-stage methods.

Comparison of detection results with L-CNN, HAWP, and F-Clip on the Wireframe dataset and the YorkUrban dataset are shown in Figure 9, respectively. Both L-CNN and HAWP rely on endpoint detection and sampling of line segment features. However, they face challenges with connectivity points and texture variations, and F-Clip predicts line segments based on angle and length, making it more sensitive to angle. This sensitivity limits its accuracy in detecting long line segments. In contrast, the method proposed in this chapter has higher detection accuracy, check-all rate, and overall performance.

Figure 9

Visualization of line segment detection methods on Wireframe dataset and YorkUrban dataset. (a) Label, (b) L-CNN, (c) HAWP, (d) F-Clip, and (e) Ours.