Detection of abnormal tourist behavior in scenic spots based on optimized Gaussian model for background modeling

3.1 Hybrid Gaussian background model based on K-means clustering optimization

In recent years, with the increasing number of stampede incidents among tourists in scenic areas, large-scale crowd behavior analysis has become a research hotspot in the field of video surveillance. The existing anomaly detection algorithms have poor recognition performance due to the interference of scenic background. The dynamic changes and complexity of the scenic environment, such as weather changes, light and shadow changes, and changes in pedestrian flow, may make it difficult to accurately update the background model. For example, the dynamic process of tourists gathering and dispersing, or the continuous changes in natural elements such as trees and water surfaces, may interfere with the background model. Lighting conditions are a very important influencing factor in the environment of scenic spots. The lighting conditions vary greatly at different times, seasons, and weather conditions, which may lead to changes in the brightness and contrast of images, thereby affecting the accuracy of anomaly detection. Therefore, based on the background subtraction method, a background model optimized by GMM is proposed to eliminate background interference. As a commonly used motion target detection method, BSM is an inter-frame differencing method that utilizes a background model to update the background image, which is obtained by differencing the background image from the main object, with the advantage of being sensitive to the background and speeding up the computation. Figure 1 displays the BSM schematic diagram.

Figure 1

Schematic diagram of BSM.

The image is binarized according to BSM as in equation (1).

where I b k ( x , y ) is the background image, I f ( x , y ) is the current frame image, D k ( x , y ) is the difference image, R k ( x , y ) is the binarization result, and T is the binarization setting threshold. GM is divided into two types according to the background model, i.e., unimodal and multimodal, and unimodal is used for the background image when the gray scale distribution is concentrated, and multimodal is used for the more dispersed time [15]. The unimodal GM and hybrid multimodal GM distribution curves are shown in Figure 2.

Figure 2

The unimodal GM and hybrid multimodal GM distribution curves. (a) Single Gaussian model distribution curve and (b) multimodal Gaussian model distribution curve.

In Figure 2, the single Gaussian distribution model curve disperses with different parameters and cannot fit well with the pixel distribution model. The mixed Gaussian distribution model combines the properties of a single Gaussian distribution model and has robust and real-time detection effects. Due to the complexity of the background in scenic areas and the uneven grayscale distribution, the study used mixed Gaussian for background modeling. The hybrid GM is modeled as in equation (2).

where K denotes the number of Gaussian distribution (GD), w i , t denotes the i th GD weight at the moment of t , X t = ( x t r , x t g , x t b ) T denotes the color vector value of the pixel point at the moment of t , μ i , t denotes the mean vector of the i th GD at the moment of t , ∑ i , t denotes the covariance matrix of the ith GD at the moment of t and ∑ i , t = δ n 2 I , I denotes the unit matrix [16]. If the three components of the image R, G, B are independent of each other, then the Gaussian probability density function is as in equation (3).

where ρ denotes the Gaussian probability density function and n denotes the pixel component dimension with a value of 3. The GM first needs to perform parameter updating at the time of foreground target extraction to estimate the GD that represents the background as well as the motion target segmentation [17]. The algorithm flow is shown in Figure 3.

Figure 3

Algorithm flowchart of mixed Gaussian model.

3.2 Optimization of GMM based on K-means clustering

In Gaussian background model, background training of Gaussian background model is required, but human objects are almost always present in many videos and human targets have a great impact on the training of the model, so the study proposes an online K-means clustering algorithm for initialization of GM. This method is able to cluster the image sequence pixel gray values, update the correlation coefficient in the clusters, and construct the GD clustering number, thus improving the speed of the algorithm. The specific optimization process is as follows, first, at the moment t = 1 , the pixel point ( x , y ) of the first frame image of a certain video is defined as in equation (4).

where m t k denotes the samples in the k th cluster at the k th moment, μ t k denotes the mean RGB color vector value, and B K t = ( b k t k , r , b k t k , g , b k t k , b ) denotes the RGB color vector value of the reconstructed background. Each pixel point’s brightness value in the newly collected image frame is categorized according to equation (5).

where ε denotes the threshold value that determines whether the pixel point is changed or not. The pixel point is categorized as class i and the update parameters are as in equation (6).

If the brightness value does not satisfy equation (6), it means that the pixel point does not belong to all the classes set, and a new class needs to be added. Additionally, the new class parameter’s starting value is shown in equation (7).

where σ t 2 denotes the variance. After X t clustering is completed, the background pixel points are updated as in equation (8).

After completing the clustering for each pixel brightness value in the training sequence N frame image, the reconstructed background image was utilized to parameterize the GM according to equation (9).

where I denotes that the unit matrix is 3 × 3. In summary, the study used K-means clustering to optimize the Gaussian model and established a background model based on a mixed Gaussian model. I hope to continuously improve the regulatory system for scenic spots in the tourism market, identify various violations and dangerous behaviors, provide important safety management references, and improve timely management efficiency.

3.3 AB recognition of tourists in tourist attractions combined with background models

Tourist attraction is a kind of regional place mainly for tourism and related activities, usually for tourists to visit and tour, vacation, fitness, etc. in their leisure time, but also with related facilities and provide corresponding tourism services in an independent management area [18]. AB refers to an unusual action that occurs in a normal position at a normal moment in a certain scenario, or an action that occurs in a non-normal position at a non-normal moment. The behavior is usually emitted by human beings, including some violent nature behaviors, which usually occur under the condition of more than two people and the action occurs fast, and some other nonviolent nature behaviors, which are usually initiated by one person and the action occurs over a longer period of time. Due to the abovementioned reasons, ABs need to be identified and monitored to provide safety technical support for tourist attraction managers [19]. The study proposes a background model based AB recognition method, and further designs the ABD method by optimizing the background model established by hybrid GM. Deep learning algorithms are excellent algorithms in the field of machine learning, which have significant development prospects in the field of image recognition and speech recognition, and can be used for recognizing pedestrians’ ABD, but there may be false detection in a certain frame or multi-frame images, and it may lead to ineffective tracking of the target on the scene with high crowd density, etc. The study proposes a detection target description method combining spatial block detection and motion foreground effect map features based on the background model established based on optimized GM, which is applied to detect tourist crowd ABD.

3.3.1 Description of detection objects in scenic area monitoring

In order to characterize the detected objects in scenic surveillance so as to locate the anomalous objects, the study is carried out in two phases, i.e., global anomaly detection and partial ABD. First, the video frames are viewed as a whole, and only the image frames of the anomalous objects are detected. Then, the video frame is segmented into equal blocks of airtime and the corresponding features are extracted separately to detect the anomaly object. The video frame is split up into several spatial blocks of the same size, using the space-time block as the detection object. Subsequently, the motion foreground effect map feature extraction is executed to derive the feature extraction frame flow, as illustrated in Figure 4.

Figure 4

Block diagram of feature extraction for sports foreground effect maps.

The motion foreground block is obtained by first extracting the background region using the optimized hybrid GM, then obtaining the spatial block and foreground image, preprocessing the two to obtain the effect weight vectors, and finally averaging the effect weight vectors to obtain the feature description. The most effective way to analyze AB is to analyze the motion information of moving pedestrians, and generally AB will show a rapid change in the motion information [20]. The background model established by the study based on the optimization GM excludes the influence and interference brought by the scenic background on the motion foreground information, so the study focuses on the tourists’ abnormal motion information here. In the extraction stage of the motion foreground block, the null partition model is set, and the conditions for the selection of the motion foreground block are as in equation (10).

(10) G i = B j , if b j L × L > λ ,

where b j denotes the number of foreground points within a block, G i ( 1 ≤ i ≤ S ) denotes that image block B j can be used as the ith motion foreground block when the condition is satisfied, L × L denotes the size of each spatial block scale, and λ is the foreground point scale threshold [21]. To describe the motion foreground block, a dense OF algorithm is used to calculate the OF vector of each pixel point of each frame of the original image sequence, which is able to provide an intuitive representation of the motion state. The study takes the average of the OF vectors of all pixel points of the motion foreground block as the current OF vector, as in equation (11).

(11) g i = 1 J ∑ j o i j ,

where g i denotes the OF vector of the ith motion foreground block, J is the all pixel points within that foreground block, and o i j denotes the OF vector of the j th pixel point in the ith motion foreground block. The definition of two indicator variables is given in equation (12).

(12) γ i j d = 1 , if dist ( i , j ) < δ d 0, else γ i j d = 1 , if − π 2 < θ i j < π 2 0, else ,

where the angle between the vector from the foreground block to the spatial block and the OF of G i is represented by θ i j , the distance threshold is represented by δ d , and the Euclidean distance between the moving foreground block G i and the spatial block B i is shown by dist ( i , j ) [22]. These two indicator variables quantify whether the spatial block is inside the sphere of impact of the foreground block. And the foreground block motion model is shown in Figure 5.

Figure 5

Motion model diagram of foreground block.

The effect weights of the motion foreground blocks on the spatial blocks are calculated as in equation (13).

(13) w i j = γ i j d γ i j θ exp − dist ( i , j ) ∥ g i ∥ , if G i ≠ B j γ i j d γ i j θ exp − 1 ∥ g i ∥ , else ,

where w i j represents the effect weights. There is a change rule between the operation foreground block G i and space block B i . Draw the effect weight values of spatial blocks based on the quantization direction of optical flow in the foreground block, as shown in equation (14).

(14) q ( ∡ g i ) = k i , if ( k i − 1 ) × 2 π p < ∡ g i ≤ k i × 2 π p h j = ( k i ) = ∑ w i j , if q ( ∡ g i ) = k i ,

where in order to make the extracted features have a high and strong differentiation, the motion direction ∡ g i of G i is quantized, p is the total number of quantized direction intervals, k i denotes the quantized direction index value of the OF of the ith motion foreground block, h j denotes the histogram statistic, and h j ∈ R p . Finally, for the sake of long-time statistics of motion foreground blocks, the computational characterization of the consecutive frames of the spatial block of the null sub-chunks is carried out, and the corresponding motion foreground characterization is shown in equation (15).

(15) h ˆ j = 1 τ ∑ i = 0 τ − 1 h j + 1 , h ˆ j ∈ R p ,

where τ denotes the number of consecutive frames.

3.3.2 ABD based on motion foreground rendering features

On the basis of the above calculation of the features of the foreground effect map of crowd motion, the resulting features are clustered, and the specific process is shown in Figure 6.

Figure 6

Flow chart for ABD based on motion foreground effect map features.

The study used an improved clustering algorithm with optimized clustering centers for its clustering, first, the mutual distances between all the samples in the sample set were calculated, and the sample point with the largest distance was selected as the clustering center [23,24,25]. Once the optimal clustering category has been identified, the collection of clustering centers is obtained by extracting the set of feature samples of all airtime blocks using the trained clustering model. After counting the number of abnormal airtime blocks, the AB is found by measuring the distance between the feature samples of the detected airtime blocks and the features of the motion foreground effect map.

4.1 Experimental parameters and process design

The background model constructed based on optimized GMMs and subsequent anomaly detection methods need to consider multiple factors: in terms of computational efficiency, GMM training usually involves a lot of iterations and parameter optimization. By optimizing the algorithm and selecting appropriate initial parameters, training time can be significantly reduced. The optimized GMM and related anomaly detection algorithms should have high computational efficiency. For large-scale datasets or high concurrency scenarios, parallel computing techniques can be used to accelerate processing speed. In terms of resource requirements, in order to train and optimize the GMM model and perform real-time anomaly detection, it is necessary to have certain computing resources to support the required computing load and ensure the stability and reliability of the system. Computers were used in the studies to run the algorithm tests and confirm that the backdrop model created by the optimization GM and the ABD approach for scenic tourists was effective. The computer is configured for Windows, and the CPU is configured for Core^TM i7-6400, 12.4 GHZ. The experiments are mainly carried out on the MATLAB 2016 platform, and different scenic spots are set up for the verification of tourist ABD. The data selected for the experiment come from the surveillance video clips of a tourist attraction and the behavioral database provided by a hospital, and the video clips include ten surveillance videos. After preprocessing, the data set the actual specific frame numbers of the anomalous frames in the video, as shown in Table 1. This dataset includes ten video segments covering three scenarios (two outdoor and one indoor, set as a, b, and c, where a and c are outdoor), with a total frame rate of 7,670 and a video frame resolution of 320 × 240. The video frames where the crowd starts to scatter and run are called abnormal frames, and all of these frames are labeled as abnormal frames. The comparison algorithms selected for the study are the existing Histograms of oriented optical flow (HOF), Histogram of maximal optical flow projection (HMOFP), and the corresponding metrics are selected as AUC and Equal error rate (EER).

The study conducts experiments on the detection and recognition effect of the ABD method based on different behavioral actions of tourists. Based on different types of behaviors set in the behavioral database, such as running, jumping, squatting, falling, wandering, etc., the proposed algorithm is used to conduct behavioral detection of the study, and the related recognition accuracy is recorded.

4.2 ABD data measurement and analysis

The experiment verifies the RA’s detection performance in three scenarios and displays the corresponding receiver operation characteristic (ROC) curves, depicted in Figure 7(a). Additionally, the values for the AUC and EER indices are shown in Figure 7(b). In Figure 7(a), the area under the ROC curve in scenarios a and c is significantly larger than in scenario b, corresponding to the AUC of 96.55 and 97.40% in Figure 7(b), respectively. It can be seen that the detection performance in scene a and scene c is better, while the detection performance in scene b is relatively weak, indicating that the research method has better outdoor anomaly detection performance.

Figure 7

ROC curve, AUC, and EER index values of the proposed algorithm in three different scenarios. (a) ROC curves for in three scenarios of UMN dataset and (b) the AUC and EET indicator values of the proposed algorithm in three scenarios.

Table 2 displays the results of a study conducted to further validate the efficacy of the designed ABD method and examine the impact of various algorithms in various scenarios with varying performance indicators. The indicators were compared with RAs were AUC and EER. The AUC and EER of RA in scenarios b and c are 88.14, 18.24, 98.12, and 7.55%, respectively, which are obviously better than the two algorithms in comparison, although in scenario a, it is 97.31 and 6.29%, which is slightly inferior to the HMOFP algorithm’s 95.18 and 12.69%, but it is expected to be very close to the ideal value, and overall has a good performance.

In the experiment, walking is set as normal behavior and others are AB, such as running, bending to walk, and falling. The corresponding data actions are collected for training to obtain the corresponding model. The backgrounds of the three traveling scenes are segmented for recognition and extraction to build the corresponding background models. In addition, the RA and Song proposed a single image crowd counting and ABD method based on multi-scale GAN networks (compared with Algorithm 1), Wu and the team proposed a new indoor effective ABD framework (compared with Algorithm 2), and Jiang et al. proposed an optimized model based on SMO algorithm and an artificial intelligence motion combination training behavior recognition model (compared with Algorithm 3) for recognition accuracy comparison, as shown in Figure 8. It can be seen that the recognition accuracy of algorithms 1, 2, and 3 is generally lower than that of the RA, with an average recognition accuracy of 89.32, 89.80, 88.23, and 93.27%, respectively. When identifying hovering actions, the accuracy of the RA and the comparison algorithm are 87.69, 84.02, 81.47, and 80.16%, respectively. This is because hovering actions are no different from normal actions, except for the length of action repetition. It can be seen that the algorithm proposed this time has good accuracy in identifying abnormal actions of tourists.

Figure 8

AB identification results.

As shown in Figure 9, in order to visually analyze the different features between AB and normal behavior, the study plotted the feature histograms of the two actions under different mode operators, and the normal behavior study selected the walking action, and AB selected the bending action. The target area contained in the image is relatively large compared to the background area, and there is a certain difference in grayscale between the background area and the target area. The peak in the histogram in Figure 9 is not obvious, indicating that the operator’s influence on the reflection of the histogram is not deep. Figure 9(a) and (b) show the histograms of the features of the stooping action under the two modes, and the stooping action shows flatter fluctuations. Figure 9(c) and (d) shows the histogram of the features of the walking action in the two modes, and the performance of the histogram of the features of the walking action fluctuates significantly.

Figure 9

Characteristic histograms of two actions under different mode operators. Histogram of bending behavior in (a) Mode 1 and (b) Mode 2. Histogram of walking behavior in (c) Mode 1 and (d) Mode 2.

Figure 10 shows the aspect ratios of the minimum outer rectangular box obtained by morphological analysis after GMM segmentation of their motion foreground maps for normal walking and running actions under the color model. In Figure 10(a), the change in the aspect ratio of the rectangular box is more balanced and regular in the case of normal walking, and in Figure 10(b), the aspect ratio of the rectangular box in the case of running is a faster change with a short duration.

Figure 10

The aspect ratio of the minimum bounding rectangle in the foreground image of normal walking and running movements. (a) Normal walking and (b) running movements.

In addition, in order to compare the performance with the most advanced ABD technology, the experiment will iterate the RA with the latest ABD algorithm to obtain the convergence curve and obtain the accuracy of ABD, as shown in Figure 11. It can be seen that the convergence curve of the algorithm proposed in the study is stable, and finally stabilizes with the ABD algorithm at an objective function of around 2.51. The ABD algorithm has a fast convergence speed and a faster decrease in the objective function. Based on the accuracy, the proposed algorithm and ABD algorithm are 0.92 and 0.88, respectively, indicating a higher ABD accuracy of the RA.

Figure 11

Comparison of convergence speed and accuracy with the latest ABD technology. (a) Iterative convergence curve and (b) tourist ABD accuracy.