Video face target detection and tracking algorithm based on nonlinear sequence Monte Carlo filtering technique

3.1.1 Fundamentals

SMCF is a Bayesian estimation through nonlinear and nonparametric Monte Carlo simulation, using samples rather than a description of probability density through functional form. The basic idea is to obtain an optimal estimate of the state using a weighted sum of a series of random samples to represent the desired posterior probability density, and the samples used are one possible description of the target state. When the number of samples increases to infinity, the Monte Carlo simulation property will be equivalent to the posterior probability density function representation, so that the filtering accuracy approximates the optimal estimation. The description of it in the mathematical context is as follows.

Let { X 0 : k i , w k i } be the requested posterior probability density p ( X 0 : k ∣ y 1 : k ) of a random observation, and let { X 0 : k i , i = 0 , … , N s } be a set of support points. The corresponding weight to the point set is { w k i , i = 0 , … , N s } . X 0 : k i = { x j , j = 0 , … , N s } denotes the set of states at the time 0 ∼ k with normalized weights ∑ w i i w k i = 1 . Thus, the posterior probability density at the moment of k can be approximated as Eq. (1).

The results can be approximated with an accuracy guaranteed by the theory of strong number laws. The outstanding advantage of the above method is that it is no longer limited by linear and Gaussian distributions and is in principle applicable to any nonlinear system that can be represented by a state space model. In addition, the method is also capable of parallel processing because of the independent homogeneous distribution of the collected samples. Compared with other forms of recursive Bayesian filters, the method is also flexible, easy to implement, and has high accuracy of approximation results.

3.1.2 SMCF technology implementation framework

Particle filtering is a practical algorithm for solving Bayesian probability, also known as condensation algorithm: bootstrap filtering, interacting particle approximation, sequential Monte Carlo methods, and so on. A particle is a filter of very small scale, which can be thought of as a point representing all possible states of the target. Filtering refers to the current state of the target that can be “filtered,” and in estimation theory, it also refers to the current state of the target estimated by the current and previous observations. The meaning of the particle filter is that the posterior probability of the propagation of the target state can be approximated by several particles. The particle filter can be applied to any nonlinear system which can be represented by state space model and the nonlinear system which cannot be represented by traditional Kalman filter. The accuracy can approximate the optimal estimation. The particle filter method is very flexible, easy to implement, has a parallel structure, and is quite practical. The computational load of recursive Bayesian filtering based on Monte Carlo simulation method is greater than that of Kalman filter and other mathematical equation solving forms. However, particle filter can be implemented in parallel. With the increase in computer speed and the emergence of parallel computers, particle filter has more practical value than traditional Bayesian filters (Kalman filter and mesh filter). By analyzing the tracking problem, a more efficient and faster particle filter algorithm can be constructed.

The SMCF algorithm typically utilizes multiple features to achieve target tracking. These features can be divided into the following categories:

1. Color features: These features typically include color information of the target area, such as color histograms, color channel histograms, etc. Color features are very useful for tracking color targets and can help algorithms distinguish between targets and backgrounds.

2. Texture features: Texture features describe the texture information of the target area, such as grayscale co-occurrence matrix, LBP, Gabor filter response, etc. These features help the algorithm recognize the texture characteristics of the target surface.

3. Spatial features: Spatial features describe the position and shape information of the target in the image. This includes the center position, scale, direction, etc., of the target. These features can be used to determine the position and attitude of the target.

4. Deep features: Deep features typically come from the middle layer of CNN or other deep learning models. These features can extract high-level semantic information of the target, such as shape, texture, color, etc.

The SMCF algorithm usually comprehensively utilizes these different types of features to improve the performance of target tracking through feature fusion or feature selection. This multi feature fusion method helps to overcome challenges in different scenarios and conditions, making the algorithm more adaptable and robust. Feature engineering and feature selection are important components of the SMCF algorithm to ensure the selection and utilization of the most informative features for efficient target tracking. Based on the above discussion and analysis of the basic principles of SMCF technology, the basic filtering steps can be summarized as follows:

1. Initial sampling: The initial moment is k = 0 , according to the prior distribution p ( x 0 ) , the initial state sample set { X K i } i = 0 , … , N ∼ p ( x k ) is established.

2. Importance sampling: At the moment k > 0 according to the expected distribution q ( x k ∣ x k − 1 j , z k ) and the collected sample X K i that makes X K i ∼ q ( x k ∣ x k − 1 j , z k ) .

3. Calculation and normalization of weights. The weights of each sample are calculated and normalized using Eqs. (2)−(4).

   (2) w k i ∝ p ( z k ∣ x k i ) p ( x k i ∣ x k − 1 i ) q ( x k ∣ x 0 : k − 1 i , z 1 : k ) ,

   (3) w = ∑ i = 1 N w k i ,

   (4) w k i = w k i w .

4. Resampling: According to the formula N eff = 1 ∑ i = 1 N w k i 2 , find out N eff , and if N eff ≥ N th , go to step 2, otherwise, execute the resampling strategy to get a new sample set { X ∼ K i , 1 / N } ∼ { x k i , w k i } .

5. Let k = k + 1 , return to step 2.

3.2.1 Sample construction and training

In this study, the sample training and construction of target tracking are done through online update. Most discriminative tracking algorithms use random sparse sampling for samples, which can neither extract effective features nor lead to a large number of overlapping samples and too few samples. In this study, a series of image blocks obtained by circular offset matrix are used as training samples, and the problem solution is converted to Fourier transform domain by the characteristics of circular matrix, which can save the sample extraction time by avoiding the matrix inversion operation [8].

As an example, a one-dimensional target vector is represented by an n*1 vector of tracking targets, denoted as x = ( x 1 , x 2 , x 3 , … , x n ) , as in Eq. (5).

The set of samples is obtained from the base sample x and the circular offset X = { P u x ∣ u = 0 , … , n − 1 } .

For two-dimensional images, the training samples can be constructed by cyclically shifting the horizontal and vertical coordinates respectively, thus having x generate a cyclic matrix, as in Eq. (6).

In Eq. (6), the first row is the basic sample and the whole matrix is all samples, obtained by circular shifting of the basic samples.

Sample training is a kernel ridge regression problem, which is actually a modification of the least squares method. The ultimate goal of training is to find a regression function that can make predictions on the sample. In constructing the sample, round-robin shift sampling is performed around the target region, and then a sufficient number of samples are generated while obtaining a computationally fast model [9,10].

Linear regression is generally expressed as f ( w , x ) = w T x , treating the sample training problem as a minimized regularized risk optimization problem, as in Eq. (7).

where λ is the regularization parameter, which is used to avoid overfitting. In this study, we take λ = 0.01 , and y i is the sample label, which is Gaussian distributed at the center of the target to ensure the maximum probability at the center.

The solution of Eq. (7) is given by

The circular matrix can be diagonalized by the discrete Fourier transform matrix as in Eq. (9).

In Eq. (9), C ( x ) is the circular shift function.

Eq. (8) can be changed as follows:

Introducing the kernel function ψ(x), w is expressed as a linear combination of x for the sample of Eq. (11).

Let the product between different samples be the result of the combination of kernel functions, as in Eq. (12).

The final regression function becomes Eq. (13).

Therefore, the solution of the kernel function-based ridge regression is Eq. (14).

where K denotes the kernel function cycle matrix, K = C ( k xx ) , and k xx is the first row of the kernel matrix K, as shown in the Gaussian kernel function Eq. (15). Gaussian kernel function, also known as radial basis function, is a kernel function used in machine learning algorithms such as SVM. It is very useful in mapping data from input space to high-dimensional feature space and is commonly used for nonlinear classification and regression tasks.

Eq. (15) is called kernel correlation filtering, where σ 2 is the dot product operation between elements, F − 1 is the Fourier inverse transform, and the solution of α in the frequency domain is α ˆ = y ˆ k ˆ zz + λ .

3.2.2 Location estimation

In face tracking, it is necessary to determine the position of the face in the next frame based on the information of the given previous frame. The response map is obtained by convolving the trained template with the candidate region as shown in Figure 1, and the response value with the largest value is the center of the face [11]. Convolutional operation is a commonly used technique in deep learning to extract feature information from input data. In object detection and image segmentation tasks, CNNs are typically used to generate response maps to locate objects or bounding boxes of interest. The following describes the process of obtaining a response map by convolving the trained template with candidate regions:

1. Select template (convolutional kernel): First, you need to define a template for detecting targets or specific features. This template is usually a small two-dimensional weight matrix, which can be manually designed or learned through training. The size and shape of the template are usually related to the target or task to be detected.

2. Prepare candidate regions: For target detection or image segmentation tasks, a set of candidate regions needs to be generated. These candidate regions are usually rectangular or non-rectangular regions on the image, which may contain objects of interest. Candidate regions can be generated through different methods, such as selective search or region based CNNs.

3. Convolution operation: For each candidate region, perform a convolution operation on the template and that region. The convolution operation actually involves multiplying the pixel values of the template and candidate regions by each element, and then summing the results.

4. Response map post-processing: After obtaining the response map, post-processing steps are usually required, such as applying non-maximum suppression to eliminate overlapping candidate regions, and setting thresholds to determine the final detection or segmentation results.

Figure 1

Schematic diagram of face tracking.

This process allows neural networks to locate regions or features of interest in the image and generate response maps, where pixel values represent the degree of template matching at different positions. This is one of the commonly used techniques in object detection and image segmentation tasks, which can help machine vision systems recognize and locate objects and features in images.

In the next frame of the video sequence, the candidate region of the target can be identified, and the set of detection samples z is constructed by circular shifting as well, and the response values of all samples can be represented by Eq. (16).

where K z = C ( k xz ) is the kernel correlation matrix composed of training and test samples z, and f ( x ) is the response map corresponding to the cyclic matrix samples composed of the candidate region z. Using the convolution property of the cyclic matrix F ( C ( x ) * y ) = x ˆ * ⊙ y ˆ , Eq. (17) is obtained.

From Eq. (17), the confidence response map of the candidate box can be quickly obtained. The position with the largest response value is also the position with the highest probability, i.e., the most likely position of the tracking target.

3.2.3 Retracking strategy

During the learning process of human, many people tend to unconsciously cover part of the face with their hands, which will make the face detected by the tracker often incorrect at this time, when the appearance model is more about the hand. Using the hand information for template update will lead to the failure of subsequent tracking and affect the evaluation of the learning effect. Traditional correlation filtering such as KCF lacks retracking mechanism, so when the face is no longer obscured by the collection, the tracking target will be lost [12].

In the target response map, the location with the highest response value is used as the predicted target. In case of good tracking condition, the target response value will be larger, while in vice versa condition, the response map peak will be smaller. So, the peak response size can be used as a basis for whether the face tracking fails or not.

In this study, the primary side regulated (PSR) is used to determine whether a face is lost to follow. Using the response map peak g max and the mean and variance of the other pixels in the response map excluding the 11 × 11 pixels around the peak μ s 1 and variance σ s 1 . The process is shown in equation (18).

From Eq. (18), it is known that PSR calculates the relative value of the peak value and the side value. When the face tracking situation is good, PSR is relatively large and the position where the response peak is located is the new position of the face. Conversely, the smaller the PSR is, the face is in an obscured situation. When the PSR is larger than 20, the face tracking is good, when the PSR is about 7, the face is lost or lost in a large area. When the PSR is larger than 3 and smaller than 10, it can be judged that the face tracking result is not reliable at this time.

If there is a PSR calculation such that the target is in an occluded or lost state, then the target model and template are not updated and the target needs to be re-detected.

In this study, an inverted pyramid is used to search for candidate boxes.

In tracking, the time of two adjacent frames is short, so they have continuity in time space. The small displacement of the target in these two frames indicates that the target in the current frame must be in the vicinity of the target position in the previous frame. In the case of consecutive occlusion of multiple frames, the candidate frame search range should be expanded. Regions 1, 2, and 3 do not overlap each other and are rectangular rings with the same center point. The distance from regions 1, 2, and 3 to the center increases one at a time, and the search range increases in turn. Each candidate frame is convolved with the template and the frame with PSR greater than a certain threshold is selected as the new position of the target where the current frame is located [13,14].

3.2.4 Main flow of the algorithm

The main process of the algorithm is summarized as follows.

In order to verify the feasibility and effectiveness of the proposed method, simulation experiments are carried out, respectively, under the condition of face rotation and size change, and the particle filter algorithm based on color histogram is compared with the proposed algorithm.

Input: Image sequence I t , target position P t − 1 , target scale S t − 1 , position estimation model α t − 1 , scale estimation model.

Output: Target position P t and target scale S t .

Step 1. For the first frame, the target features are first extracted, while using the circular matrix to construct training samples with 2.5 times the region of the real target scale. Sample labels y and y s are Gaussian distributed and y s is an N-dimensional vector, and the template is trained and saved.

Step 2. For t frame, a candidate frame is constructed with P t − 1 . The response map is obtained by convolving with the template, and the one with the largest response value is used as the position of the target in the current frame, and the PSR is calculated.

Step 3. If the tracking result is unreliable, the inverted pyramid method is used to search several candidate frames and correlate them with the template, and the candidate frame with PSR greater than a set threshold is selected as the target position [15].

Step 4. If the tracking result is reliable, after acquiring the new position of the target in the current frame, please extract the features and construct a feature pyramid with S t − 1 , then use the scale filter to perform the correlation operation and get the scale with the largest response value as the target scale. The scale with the largest response value is used as the target scale S t .

Step 5. According to the obtained P t and S t , the training samples are constructed with the circular matrix and the model is updated.