Accurate and real-time object detection in crowded indoor spaces based on the fusion of DBSCAN algorithm and improved YOLOv4-tiny network

4.1 Improvement of the network structure

The backbone network of the YOLOv4-tiny model contains three CSP modules, namely CSP1, CSP2, and CSP3. Among them, the CSP2 layer includes precise location information and more specific information, but fewer semantic information. The CSP3 layer includes a larger amount of semantic information but less specific information, the location information is relatively rough, and the position information and detail information of many small targets could be lost. To promote the recognition precision of the YOLOv4-tiny model for densely crowded targets, we propose an improved YOLOv4-tiny model, the detailed structure of which is shown in Figure 2. In the figure, Conv denotes convolutional operation, Leaky is the Leaky Relu activation function, and Maxpool represents max pooling operation. The route module performs concatenation operation, and CSP1, SCP2, and CSP3 are CSP network modules. UP1 and UP2 represent upsampling 1 and upsampling 2 operations, respectively. The input image resolution is 416 × 416. The dimensions of the output feature maps are 13 × 13, 26 × 26, and 52 × 52, respectively.

Figure 2

Structure of improved YOLOv4-tiny model, CBL module, and CSP module.

From Figure 2, it can be seen that the proposed model adds a path connected to the CSP2 layer of the backbone network in the neck network of the original YOLOv4-tiny model, and the detection scales are expanded from two to three. The outputs of the CSP2 layer and the upsampling layer (UP layer) are concatenated in the channel dimensions, the feature maps from two different layers are fused, and then through two CBL modules, the feature map that is downsampled by eight times of the resolution of the input image is finally obtained. For an input image with a resolution of 416 × 416 pixels, the improved YOLOv4-tiny model outputs feature maps at three scales with resolutions of 13 × 13, 26 × 26, and 52 × 52 pixels, respectively. Since the occluded objects have the characteristics of small salient areas, the original YOLOv4-tiny network is prone to lose a lot of edge detail information. In the proposed model, large-scale feature map optimization technique is introduced into the improved backbone network, which enables the network to capture more detailed image information. By improving the resolution of input pictures, the dimensions of the output feature map are changed from 13 × 13, 26 × 26 to 13 × 13, 26 × 26, and 52 × 52, thereby enhancing the learning capability of the shallow network and reducing the information loss of the shallow layers in the training process.

4.2 CmBN strategy

In object detection tasks, due to the limited memory capacity of the video cards, the BN strategy is inaccurate in estimating statistical information, which may easily lead to increased model errors. This drawback will be more pronounced in resource-constrained embedded devices and can significantly degrade model performance. To this end, the CmBN strategy is used to replace the BN strategy in the original YOLOv4-tiny model.

During model training, the batch of image data in the training set is evenly divided into several mini-batches and passed to the model for forward propagation. The weights of the model do not change until a single iteration is completed, so the statistics of different mini-batches in the same batch can be directly accumulated. The execution processes of the CmBN strategy are as follows:

1. When calculating the mean and standard deviation of the ith batch data, the output information of the convolutional layer of the (i − 1)th mini-batch data in the training set is combined;

2. The output of the convolution layer of the ith small batch of data is transformed into a normal distribution with a mean of 0 and a variance of 1;

3. Using learnable parameters, linearly transform the normalized output of the convolutional layers of the ith mini-batch to enhance its expressive ability.

The forward pass calculation process of the CmBN strategy is as follows:

where l denotes the number of convolutional layers and j denotes the index number of the mini-batch. m represents the size of the mini-batch data, i is the index number of the mini-batch data, and τ is the serial number of the mini-batch index. x j , i l is the output at the lth convolutional layer for the ith batch of data from the jth mini-batch. ς ¯ j l and ς ¯ j − τ l are the average values of the output at the lth convolutional layer for the jth and ( j − τ ) th mini-batch, respectively. v ¯ j l is the sum of squares of the mean output at the lth convolutional layer for the j-th mini-batch. φ ¯ j l is the standard deviation of the output at the lth convolutional layer for the jth mini-batch. δ is a constant added to increase numerical stability. x ˘ j , i l is the normalized output at the lth convolutional layer for the ith batch of data from the j-th minibatch. γ and β are learnable scaling and translation parameters, respectively. y j , i l is the result obtained after performing CmBN on the output of the lth convolutional layer for the ith batch of data from the jth mini-batch.

4.3 Target bounding box prediction

In the majority of DL-based target detection algorithms, the convolutional layers generally only acquire the features of the targets, and then pass the acquired features to the classifier or regressor for result prediction. The YOLO series of algorithms use a 1 × 1 convolution kernel to complete the target prediction so that the dimension of the obtained prediction map is equal to the dimension of the feature map. The number of target bounding boxes in YOLOv4-tiny is 3, which is determined by each grid cell in the prediction map, and the feature map contains b × ( 5 + C ) grid information, where C is the total number of categories and b represents the number of bounding boxes obtained, and different targets have their own unique bounding boxes. ( 5 + C ) is the parameter for each bounding box, which consists of the center coordinates, the size, the target score, and the confidence of the C classes of the bounding box. The relationship between the predicted bounding boxes and the network output can be expressed as follows:

From Figure 3, it can be seen that the predicted center coordinate is ( b x , b y ) , where b w and b h are the width and the height of the forecasted bounding boxes, respectively. t x and t y represent the offset values of the predicted target center relative to the center coordinates, and the values of t x and t y are controlled in the [0, 1] interval by the sigmoid function.

Figure 3

Schematic diagram of bounding box prediction.

The YOLOv4-tiny algorithm consists of two parts, training and testing. During the training stage, a mass of data needs to be fed into the model, and during the prediction stage, the candidate bounding boxes are used to determine whether any target falls into the candidate box. If a target falls within the bounding box, its probability is as follows:

In the process of object detection, images vary in size and variety. In order to determine the initial positions of the candidate bounding boxes in YOLOv4-tiny, the K-means clustering algorithm is utilized to determine the initial position of the bounding boxes. As an unsupervised learning method, the K-means clustering algorithm performs clustering operations on surrounding objects close to them by specifying a K value as the center. Through repeated iterations, the cluster center value is updated. When the intra-class difference is smaller and the out-of-class difference is larger, the desired effect is achieved, and the measurement of the “difference” is calculated based on the distance from the sample point to the centroid of the class to which it belongs. Euclidean distance is generally used to measure this difference, and the Euclidean distance measurement can be calculated as follows:

where x is the sample point within the class, μ is the centroid within the class, n is the number of samples in each class, and i is the mark number of each data point.

However, the appropriate selection of K value in K-means clustering algorithm is often hard to complete, and it will directly affect the effect of clustering. At the same time, the clustering results of the K-means method can only guarantee the local optimal clustering in general and are more sensitive to noise interference. In addition, the K-means clustering algorithm is difficult to achieve the ideal clustering effect for non-convex data and data with large differences in size. In practice, the number of targets and the number of object categories in an image are generally unknown, and the objects may also be distributed in a very scattered way. Therefore, if the clustering is performed according to the specified K value, the distance between the obtained center point and the actual position may be too far. In contrast, the DBSCAN clustering algorithm has no bias on the shape of the clusters and does not need to input the number of clusters to be divided, only the appearance radius of the target is required to be set. In addition, the impact of the center point being too far away from the actual location can be alleviated by performing clustering based on density-reachable concept. Therefore, this article proposes a DB-K clustering method that combines DBSCAN and K-means algorithms. First, through the DBSCAN clustering algorithm, the initial good clustering effect is obtained under the condition of ignoring the center point, and thus several completed clusters are obtained. Then multi-scale clustering is added to further obtain the local and overall information of the object; specifically, multi-scale clustering is performed on the contour information of the object. After multi-scale clustering and convolution operations, the initial feature maps can be obtained. Finally, these clusters are used as input data, and the K-means algorithm is used to divide the clusters so that the accurate center point position can be obtained. This method can effectively speed up the convergence of the dataset and promote the classification accuracy of small objects.

In the DBSCAN clustering algorithm, the closeness of the sample distribution in the neighborhood is described by the parameter ( ε , MinPts ) . Let the input sample be D = ( x 1 , x 2 , … , x m ) ; if the number of samples in the set of subsamples satisfies ∣ N ∈ ( x j ) ∣ ≥ MinPts , then the subsamples are added to the core sample set Ω = Ω ∪ { x j } . Subsequently, search for all subsample sets by neighborhood distance threshold to update the sample set of the current cluster, and the unvisited sample set will also be updated accordingly. Finally, the output result after iteration is C = ( c 1 , c 2 , … , c k ) .

In the original YOLOv4-tiny, the seed points of the K-means clustering algorithm are chosen randomly, which increases the randomness of the clustering and will lead to poor clustering effect. This article proposes a strategy to reduce the randomness of seed point selection. First, the K value is obtained through the error sum of squares method, as shown in equation (11), and then the first cluster center is obtained through the K-means algorithm. Afterward, the obtained K clusters are analyzed, and the closest clusters are merged, thereby reducing the number of cluster centers; the corresponding number of clusters will also decrease when the next clustering is performed, obtaining an ideal number of clusters. After several iterations, when the convergence of the evaluation function reaches the expected value, the best clustering effect can be obtained as follows:

In the experiment, 400 random points are selected and tested with the K-means clustering algorithm and the proposed DB-K hybrid clustering algorithm, respectively. In the K-means clustering algorithm, the K value is specified as 3, and the experimental results are shown in Figure 4(a). In the proposed DB-K clustering algorithm, the value of K is 3 and it is determined by the DBSCAN clustering algorithm, and the experimental results are shown in Figure 4(b). It can be clearly seen from the experimental results that the method proposed in this article has better clustering effect.

Figure 4

Performance comparison of different clustering methods. (a) K-means clustering method and (b) DB-K clustering method.

Therefore, by integrating the DBSCAN clustering algorithm and improving the traditional K-means clustering algorithm, the classification performance of object detection in different datasets and complex environments is enhanced, and the robustness to noise and interference is also improved. With the proposed DB-K clustering algorithm, the performance of neural network in object recognition is significantly improved.

5.1 Experiment datasets

In order to analyze the recognition performance of the proposed lightweight model in the crowded indoor spaces with occluded objects, two datasets were constructed: the PASCAL VOC07 + 12 public dataset which includes 16,551 training images and 4,952 testing images, and a self-made dataset. The self-made dataset involves four different scenarios, with 10,432 images captured from elevator cars and 9,139 images captured from bus cars, passenger aircraft cabins, and natural scenes. The training, validation, and test sets are partitioned in a ratio of 7:2:1. Among them, 70% of the pictures have the characteristics of mutual occlusion or incomplete camera framing. In this article, such pictures are defined as complex images in the experiment, which can effectively prevent the overfitting caused by the single background in the dense space during training, and improve the generalization capability of the model in crowded indoor scenarios. The dataset defines three detection categories: person, electric-bicycle, and bicycle. Baby carriage, trolley, furniture, other goods, and pets are used as negative samples to enhance the reliability of the model. The LabelImg tool designed by Tzutalin [39] is used to mark the labeling area and picture area of each picture according to a certain proportion, and the aspect ratio of the pictures is limited within 3:1 to make the predicted bounding boxes more fit to the targets. The statistics of the self-built dataset is shown in Table 1.

Objects in dense indoor scenes such as elevators and carriages are prone to mutual overlap and occlusion, which will seriously affect the accuracy of target recognition. Aiming at the aforementioned problems, this study introduces Mosaic data augmentation technique [40] during model training to improve the model’s capability to recognize occluded objects. The example images after Mosaic process is shown in Figure 5. Before each iteration, the DL framework not only acquires images from the training set, but also generates new images through Mosaic data augmentation, and then the newly generated images and the original images are combined and fed into the model for training. Mosaic data enhancement randomly selects four images from the training set, randomly crops the four selected images, and splices the cropped images in sequence to obtain a new image, and the generated image has the same resolution as the training set image. During random cropping, part of the bounding box of the target in the training set image may be cropped to simulate the effect of the object being occluded. In addition, this study optimizes Mosaic data augmentation, and proposes an improved Mosaic data augmentation, which uses the intersection-over-union (IoU) as an indicator and sets the threshold considering the relevant standards used when calibrating the dataset to filter the objects’ bounding boxes in the newly generated image. The improved Mosaic strategy generates a new image according to the steps of Mosaic data enhancement and filters the object bounding boxes in the newly generated image. If the IoU between the object bounding boxes in the new image and in the corresponding original image is less than the threshold, the object bounding box in the new image will be deleted and it is considered that there is no object here. Otherwise, the object bounding box in the newly generated image will be retained, and it is considered that there is a target here.

Figure 5

Examples of the training data enhancement. (a) Original images and (b) imaged after enhanced Mosaic process.

5.2 Evaluation metric

The inference speed (ms/frame) is used as an evaluation metric for model recognition speed, and the inference speed is the time required for the model to recognize an image. The average precision (AP) is used as the evaluation metric for the model recognition accuracy. AP is calculated based on the precision (P) and recall (R) of the model. P, R, AP, and mAP are calculated as follows:

where TP is the number of correctly detected positive samples, TN is the number of correctly detected negative samples, FP is the number of falsely detected positive samples, and FN is the number of falsely detected negative samples. P represents the percentage of correctly detected positive samples in all detected positive samples. R represents the percentage of correctly detected positive samples in all of the ground-truth positive samples. In the final evaluation result, AP represents the comprehensive evaluation of a certain category, and the greater the AP value is, the better the accuracy of a single category. mAP is an evaluation of the level of the entire network, C is the number of classes contained in the whole dataset, and c denotes a single class.

5.3 Model training parameters

The model parameters in the experiment are set as follows: the input image is 416 × 416; the epoch is set to 300; the batch_size is 128 for the first 70 rounds, and 32 for the last 230 rounds. The learning rate is 1 × 10^–3 for the first 70 rounds and 1 × 10^–4 for the last 230 rounds. We employed stochastic gradient descent optimization strategy for model training, with an initial learning rate whose momentum was gradually decreased as the degree of gradient descent increased, in order to improve convergence performance. Specifically, we set the momentum parameter to 0.9 to accelerate the convergence rate of the optimization process. The loss curves during training are shown in Figure 6.

Figure 6

Loss curves during training.

As the epoch continues to increase, the loss values of both modes continue to decrease. After 70 rounds of training, the loss curves tend to be stable, and there is no underfitting or overfitting. The loss values of the original YOLOv4-tiny algorithm and the improved algorithm converge to around 2.3 and 1.9, respectively, which proves that the recognition accuracy of the model is constantly improving, and the hyperparameters of the proposed algorithm are set reasonably.

5.4 Comparison with the original YOLOv4-tiny

The test results of the original YOLOv4-tiny and the proposed model on self-built dataset are shown in Table 2, in which the mAP, inference speed, and model size comparisons are given. Among them, complex images account for 70% of the dataset, which correspond to the situation where the targets are occluded by each other or the camera view is incomplete. The mAP of the original YOLOv4-tiny model for recognizing complex images and all images in the test dataset are 75.47 and 84.25%, respectively, and the proposed model is 7.94 and 8.32% higher than those of the original model, respectively. The neck network of the improved YOLOv4-tiny model reduces the loss of low-level feature map details and location information in the backbone network. The newly added feature maps have a smaller perceptual field for detecting blurred and occluded targets in images and improve the model’s ability to detect small targets. And, the DB-K clustering algorithm is used to further improve the classification effect. Compared with the original Yolov4-tiny, the model size of the proposed model is increased by 2MB, and the inference time per image is increased by 0.37 ms. The improved model only adds three convolutional layers, one upsampling operation, and one connection operation to the neck network, and the backbone network of the model does not change. Therefore, the improved YOLOv4-tiny model maintains a fast inference speed while improving the detection accuracy.

The mAP curves of the proposed model and the original YOLOv4-tiny on PSACAL VOC dataset are shown in Figure 7. From the figures, it can be observed that with the proposed model, the mAP for recognizing objects of various scales and different categories are improved significantly. For example, compared with the original model, when using the proposed model, the APs for categories containing a large number of small targets such as potted plant, boat, and bird are increased by 7, 4, and 3%, respectively. The results validate that the proposed algorithm has better detection effect when dealing with scenes containing many occluded targets and small targets, which further proves the superiority of the proposed algorithm.

Figure 7

Performance comparison on PASCAL VOC dataset. (a) Original YOLO-v4-tiny and (b) proposed model.

5.5 Ablation study

The ablation analysis is performed based on the original YOLOv4-tiny model and combined with different improvement strategies, and the training and performance evaluation are carried out on the self-built dataset containing four types of scenes, to validate the contribution of each module to the improvement of recognition accuracy under the precondition of guaranteeing real-time performance. The test results of different module combinations are shown in Table 3, all of which are trained using the proposed enhanced Mosaic technology. The effectiveness of each module is analyzed in the table, and it can be seen that each improved module has different degrees of contribution to the overall performance. Among them, the introduction of DBSCAN in Model 4 contributes the most to the network, and the mAP increases by 4.99%, which proves that the proposed clustering algorithm significantly improves the classification effect in complex environments, while increasing the robustness to noise and interference. The mAP value of the original YOLOv4-tiny (model 1) is 84.25%, and the mAP value after using the neck network optimization (model 2) is 86.77%, an increase of 2.52%, indicating that the introduction of this optimization module enables the model to strengthen the extraction of detailed information from shallow layers, which makes the training for occluded objects more in-depth. Model 3 replaces BN with CmBN on the basis of model 2. BN can only use the output features from the convolution layers of the current mini-batch, so the statistic information is not accurate enough, resulting in a poor performance. CmBN, on the other hand, realizes the expansion of samples by accumulating information from different mini-batches and makes the estimation of statistical information more accurate.

5.6 Comparison with other state-of-the-art models

To further verify the effectiveness of the proposed model, a comparison experiment is carried out on PASCAL VOC07 + 12 public dataset with the proposed model and other state-of-the-art models including Faster RCNN [30], YOLOv4 [36], YOLOv3-tiny [34], and YOLOv4-tiny [10]. In addition, the DL frameworks and the trained models are ported to Jetson nano and Raspberry Pi 4B to test the inference speed of the models on the embedded hardware platforms. The results are listed in Table 4. From Table 4, it can be observed that the advantage of large networks is high detection accuracy. For example, the two-stage algorithm Faster RCNN using Resnet50 as the backbone network and the classic one-stage algorithm YOLOv4 have achieved mAPs of 83.77 and 90.01%, respectively. However, the size of these two models is too large, 330 MB and 256 MB, respectively, making such networks difficult to deploy to mobile terminals with limited computing capacity. The performance of the proposed method is comparable to that of the large network Faster RCNN in mAP, only 1.41% behind, and the size of the proposed model is only 1/13 to that of the faster RNN. Compared with the 64.3 million parameters of the YOLOv4 model, the parameters of the proposed model are only 6.1 million, which is less than 1/10. The advantage of the lightweight network is that the detection speed and accuracy are relatively balanced, and it can perform real-time detection on the mobile terminal, but the performance is relatively poor in complex scenes. Compared with two popular lightweight models YOLOv3-tiny and YOLOv4-tiny, the proposed method has significantly improved mAP performance while satisfying the real-time detection requirements. The proposed model will not introduce too much extra computing power and memory overhead in the inference process. The model retains the advantages of simplified structure and fast inference while ensuring a high recognition accuracy. It proves that the proposed model is applicable for the deployment in embedded hardware platforms.