Indoor environment monitoring method based on the fusion of audio recognition and video patrol features

3.1 Abnormal behavior video feature detection based on CNN-ConvLSTM network structure

CNN is a commonly used method of DL technology in image recognition. CNN extracts feature information by processing images in convolutional, pooling, and fully connected layers. AlexNet is an optimized neural network based on CNN, mainly used in image recognition. AlexNet consists of three different layers, namely, convolutional, pooling, and fully connected layers [12]. Each level uses a different learning rate and loss function to learn image features to achieve image classification or recognition. This study utilizes CNN optimized AlexNet for video FE, and the FE is denoted in Figure 1.

Figure 1

AlexNet under the video FE flow chart.

Figure 1 is the flowchart of video FE using the AlexNet. Traditional FE networks directly take the original video frames as input. However, since using adjacent images as input to the network can improve the FE effect, this study first performs subtraction processing on the input images, and then uses the AlexNet for FE. In the AlexNet, it includes convolutional, pooling, and fully connected layers, respectively. To improve the training speed of the network, this study adds a ReLU function after each convolutional layer, thereby increasing the sparsity of the network and improving its training speed. Local response normalization is required after each pooling layer to avoid information loss during the extraction.

Due to the temporal nature of video features, using only the AlexNetcan only extract action features at each moment, without continuity. This study builds a temporal signal model using LSTM based on the AlexNet, aiming to assist the AlexNet in extracting features with temporal variations. LSTM is different from traditional CNN. LSTM can observe and record the changes in continuous video frames, and can also avoid the problem that each sample has a different time length. To enable LSTM to characterize local features through convolutional operations like CNN, this study uses CNN to replace the fully connected layer in LSTM and constructs the final temporal signal model using a ConvLSTM network. The calculation method for the ConvLSTM input gate is shown in Eq. (1) [13].

where i t expresses the input gate at time t ; W i indicates the input gate weight; ∗ denotes convolution operation; σ means the sigmoid function; b i denotes the threshold of the input gate; h t − 1 represents the output of the previous layer of network at time t − 1 ; and x t denotes the input of the local network at time t .

where f t expresses the forgetting gate at time t ; W f stands for the forgetting gate weight; and b f indicates the threshold of the forgetting gate.

where O t indicates the output gate at time t ; W O refers to the output gate weight; and b O stands for the threshold of the output gate.

where C t stands for the memory unit at time t ; W C refers to the weight of the memory unit; b C means the threshold of the memory unit; and tanh denotes a hyperbolic tangent function.

where ⊗ means Hadamard multiplication. By using Hadamard multiplication to multiply the corresponding elements, the calculation method for memory units can be obtained. C t − 1 expresses the memory unit of time t − 1 .

where h t means the output value of the local network at time t . The combination of ConvLSTM with AlexNet can obtain the CNN ConvLSTM abnormal behavior video feature detection network structure illustrated in Figure 2.

Figure 2

Structure of CNN-ConvLSTM anomalous behavior video feature detection network.

Figure 2 shows the structure diagram of an abnormal behavior video feature detection network constructed using the CNN-ConvLSTM network. In Figure 2, the entire network structure mainly contains four parts, namely, video frame differential processing, CNN, LSTM, and fully connected layers. First, it divides a video into countless frames, then performs differential processing on the adjacent two video frames, and then uses the AlexNet for FE. After FE is completed, ConvLSTM is used for temporal information modeling, and finally, a fully connected layer is applied for output.

3.2 Research on end-to-end abnormal audio feature detection based on original waveform

In addition to processing the video captured by the HD camera in the business hall, extracting the image frames in the video for identification and detection, the research further identifies and monitors the audio information in the indoor environment, aiming to judge various behaviors in the indoor environment through audio analysis, to achieve the purpose of real-time monitoring the dynamic of staff. Before analyzing a section of indoor audio information, it is necessary to first perform signal preprocessing, mainly including routine operations such as pre-emphasis, framing, and windowing. It assumes that the total duration of the audio information used in this study is 2 s and the frame length is 20 ms. The main calculation method for pre-emphasis is indicated in Eq. (7) [14].

where x ( n ) means the original audio signal sequence; n indicates the time of speech sampling; y ( n ) expresses the audio signal sequence after pre emphasis; a is the pre emphasis coefficient; x ( n − 1 ) refers to the audio signal sequence at time n − 1 .

Due to the nonlinear nature of the staff’s audio signals collected in the indoor business hall, it is necessary to perform framing processing on the audio signals to ensure their integrity is preserved during the extraction. This study adopts the overlapping segmentation method for frame segmentation, with a frame length of 20 ms and a frame shift of 10 ms. After framing the audio signal, windowing is also necessary to prevent signal distortion during the processing. Hamming and rectangular windows are two common window functions, and their expressions are denoted in Eqs. (8) and (9).

Eq. (8) is the window function of a rectangular window, where w ( n ) means a window function. n expresses the windowing parameter. N stands for the range of values and is a constant.

Eq. (9) is the window function of Hamming window, where w ( n ) stands for window function. n stands for the windowing parameter. N refers to the range of values and is a constant. Due to the loss of waveform details in the high-frequency part during the use of rectangular windows, the Hamming window can effectively ensure audio details and make the waveform more complete. Therefore, Hamming window function is selected for the windowing audio signal in this study.

Because the audio events in surveillance videos are mostly audio clips with inconsistent durations, there is a significant difference in audio features between different periods. To better extract audio features, the study utilizes Mel-frequency cepstral coefficients for FE. The Mel cepstrum coefficient can combine the auditory features perceived by the human ear with the speech generation mechanism, thereby extracting audio features that are more in line with the actual situation [15,16]. The FE of Mel cepstrum coefficients is shown in Figure 3.

Figure 3

Diagram of audio FE of Mel-frequency cepstral coefficients.

Figure 3 shows the audio FE diagram of Mel cepstrum coefficients. A segment of original audio will be preprocessed to remove redundant interference signals, and then a discrete Fourier transform will be performed to obtain the spectrum. It squares the spectrum to obtain the energy spectrum, and then filters it using M Mel bandpass filters to obtain the output power spectrum of the spectrum. Finally, the audio signal will be subject to an inverse discrete Fourier transform to obtain the static characteristics of the Mel cepstrum coefficient, and the dynamic characteristics can be obtained by differential processing of the static characteristics. Linear predictive coding can calculate different sound characteristics according to the ideal acoustic model. Combining linear predictive coding with Mel cepstrum coefficients can further determine the source and type of sound in the video. The calculation method of audio error prediction is indicated in Eqs. (10) and (11).

where p refers to the amount of linear predictive coding; x ( n ) and x ( i ) stand for different sound signals, respectively; and a i indicates the Mel cepstrum coefficient.

where x ˜ ( n ) stands for the predicted sound signal; and e ( n ) expresses prediction error.

When using Mel cepstrum coefficients to extract audio features, a series of transformations are required, resulting in Mel cepstrum coefficients only changing within a limited frequency range, making it impossible to extract all the features of the audio. Moreover, it is difficult to ensure the accuracy and stability of feature parameters during real-time processing and analysis of audio. Therefore, a DL network is further used to build an audio feature detection model, as expressed in Figure 4.

Figure 4

Structure of the original audio FE network.

Figure 4 denotes the structure diagram of the original audio FE network constructed using AlexNet and LSTM. As shown in Figure 4, a complete multimedia audio segment will first undergo audio framing processing, and then the corresponding waveform of each frame will be used as input to the AlexNet for audio FE. After AlexNet processes the audio signal, it will transmit the features to the LSTM network to construct a temporal model. The last LSTM unit input is the most effective feature of the entire audio segment, which is connected to the fully connected layer to complete the final audio feature classification and facilitate subsequent system recognition.

3.3 Abnormal behavior monitoring based on MCAVFF

Multi-channel feature fusion refers to the fusion of feature information from multiple channels, such as the frequency domain, time domain, or spatial parts of an audio signal, to generate a comprehensive feature vector to represent the audio signal. Multichannel feature fusion can be used in multiple audio processing tasks such as speech recognition, audio arrangement, and speech synthesis. To facilitate the observation of abnormal behavior of staff in the indoor environment, this research combines the video and audio detection models built above, and uses multichannel feature fusion technology to reduce the inherent defects in the process of single-channel recognition features, to further promote the accuracy of recognition and detection. For a complete piece of multimedia information, it is first necessary to perform shot segmentation processing to separate its audio content from the video content, ensuring that there is visual and auditory information in each video sequence. The process of shot segmentation processing is expressed in Figure 5.

Figure 5

Flow chart of lens segmentation.

Figure 5 shows the specific process of shot segmentation, where a complete video is first divided into countless segments through shot segmentation processing. Subsequently, the visual and audio information in these fragments will be extracted for feature processing. In the above research, the AlexNet-ConvLSTM model has been constructed for processing video features, while the AlexNet LSTM model has been constructed for processing audio features. To further simplify the FE task, the study utilizes the multichannel principle to fuse the features of the two models mentioned above, and then send them together into the LSTM network for FE. Finally, the indoor environment monitoring model using the DL network is built as denoted in Figure 6.

Figure 6

Indoor environment monitoring model with DL network.

Figure 6 shows the indoor environment monitoring model built by the final use of the DL network, where the constructed model is composed of two channels: video and audio processing channels. In the video processing channel, the AlexNet is studied to extract differential information between adjacent frames of video and identify their differential features. At the same time, on the audio processing channel, the AlexNet is studied to process the original waveform features of each frame of the audio signal, and the adjacent features at each time are fused and fed into the LSTM model for temporal modeling. To sum up, the indoor environment monitoring model finally built above can well assist managers to monitor the working status of staff in the business hall and identify their abnormal behaviors. When dealing with audio events in incongruous videos, the study adopts a separation and association approach. The video is divided into independent segments by lens segmentation technology, so that the audio and video content of each segment can be analyzed separately. In the audio processing process, the integrity of the audio signal is ensured through the application of signal processing techniques, such as pre-accentuating, framing and Hamming windowing. In feature extraction, combining with Mel cepstrum coefficient and other methods, we can not only extract the time domain and frequency domain features of audio signals, but also enhance the recognition ability of irregular audio events.

4.1 Analysis of abnormal behavior video feature detection results

To test the effectiveness of AlexNet-ConvLSTM, the raised network model was compared with traditional video feature detection models. The self-made video action dataset was selected as the validation dataset for this experiment. First, the loss values of different models were tested as a function of iteration times, as shown in Figure 7.

Figure 7

Variation in loss values of different video feature models with the number of iterations. (a) Variation in loss curve of AlexNet-ConvLSTM network. (b) Variation in loss curve of CNN network.

Figure 7(a) and (b) expresses the variation in loss values with iteration times for the AlexNet-ConvLSTM and the traditional CNN models, respectively. As shown in Figure 7(a), as the iteration times increased, the training and validation losses of the AlexNet-ConvLSTM model showed a continuous decreasing trend. When the number of iterations reached 35, the model began to stabilize. As shown in Figure 7(b), as the iteration times increased, the training and validation losses of traditional CNN models also showed a continuous decreasing trend. However, both the training loss and validation loss had significant variations. When the iteration times reached 45, the CNN model began to stabilize. The results in Figure 7 show that ConvLSTM can capture dynamic information in video data more effectively by introducing LSTM’s timing processing capability, thereby improving the learning efficiency and effect of the model.

Figure 8(a) and (b) shows the variation in detection accuracy values with iteration times for the AlexNet-ConvLSTM and traditional CNN models, respectively. As shown in Figure 8(a), as the iteration time increased, the training and validation accuracy of the AlexNet-ConvLSTM model showed a continuous upward trend. When the number of iterations reached 25, the model began to stabilize, and the detection accuracy of the AlexNet-ConvLSTM model was around 0.95. From Figure 8(b), as the iteration time increased, the training and validation accuracy of traditional CNN models showed a constantly changing trend, and the range of changes in training accuracy was much greater than the validation accuracy. When the number of iterations was 15, the detection accuracy of the CNN model remained stable at around 0.80. The results in Figure 8 highlight the effectiveness of the model in processing complex video data, indicating that it can enable efficient abnormal behavior monitoring in future practical applications.

Figure 8

Variation in test accuracy of different video feature models with the number of iterations. (a) Variation in detection accuracy of AlexNet-ConvLSTM network. (b) Variation in detection accuracy of CNN network.

4.2 Analysis of abnormal audio feature detection results

To test the performance of the audio FE model AlexNet-LSTM, the proposed network model was compared with traditional audio feature detection models. It selected a self-made voice dataset as the validation dataset for this experiment. First, it tested the loss values of different models as a function of iteration times, as denoted in Figure 9.

Figure 9

Variation in loss values with the iteration times for different audio feature models. (a) Variation in loss curve of AlexNet-LSTM network. (b) Variation in loss curve of LSTM network.

Figure 9(a) and (b) show the variation in loss values with iteration times for the AlexNet LSTM and the traditional LSTM models, respectively. As shown in Figure 9(a), as the iteration times increased, the training and validation losses of the audio FE model AlexNet LSTM showed a continuous decreasing trend. When the iteration times reached 60, the model began to stabilize. As shown in Figure 9(b), as the number of iterations increased, the training and validation losses of traditional LSTM models also showed a continuous decreasing trend. However, the training loss of LSTM varied significantly and there were fewer coincident trajectories with the validation loss. When the iteration time was 80, the LSTM model tended to stabilize.

Figure 10(a) and (b) shows the variation in detection accuracy values with iteration times for the AlexNet LSTM and the traditional LSTM models, respectively. As shown in Figure 10(a), as the number of iterations increased, the training and validation accuracy of the AlexNet-LSTM model for audio datasets showed a continuous upward trend. When the iteration times reached 50, the model began to stabilize, and the detection accuracy of the AlexNet-LSTM model was around 0.97. From Figure 10(b), as the iteration times increased, the training and validation accuracy of traditional LSTM models showed a constantly changing trend, with significant changes. When the number of iterations was 60, the detection accuracy of the LSTM model remained stable at around 0.90. The analysis of Figures 9 and 10 further reveals the performance of the audio feature extraction model, and the convergence and detection accuracy of AlexNet LSTM model are significantly better than the traditional LSTM model. This result not only highlights the importance of deep learning in audio signal processing but also highlights the potential for models to be used in real-time monitoring and responding to emergencies, thereby providing better analytical tools for related fields.

Figure 10

Variation in test accuracy of different audio feature models with the number of iterations. (a) Variation in detection accuracy of AlexNet-ConvLSTM network. (b) Variation in detection accuracy of LSTM network.

4.3 Analysis of detection results of the MCAVFF model

The above research proved that the video and audio feature detection models, AlexNet-ConvLSTM and AlexNet-LSTM, performed well in both video and audio FE. To further demonstrate the excellent FE ability of MCAVFF models, a comparative study was conducted on their detection results.

Table 1 shows the detection accuracy, recall rate, F1 value, and mean average precision (MAP) values of CNN, LSTM, and multichannel audio video feature fusion models in a single video dataset, a single sound dataset, and a video sound mixed dataset, respectively. From Table 1, the detection accuracy, recall, F1 value, and MAP value of the proposed model in the video-audio dataset were 97.85, 96.98, 95.61, and 35.61%, respectively. In summary, the MCAVFF model performed better than traditional CNN and LSTM models under various indicators.

Figure 11 shows the satisfaction of users and experts with various FE models. Figure 11(a) shows the satisfaction of users with various FE models. From Figure 11(a), the user satisfaction levels for CNN, LSTM, and multi-channel fusion models were 0.76, 0.78, and 0.92, respectively. Figure 11(b) shows the satisfaction of experts with various FE models. From Figure 11(b), experts’ satisfaction with the three models was 0.72, 0.82, and 0.97, respectively. The positive feedback from users and experts further verified the performance and practicability of the model, indicating that it effectively met the needs of practical applications and provided reliable technical support for indoor environmental monitoring and safety management.

Figure 11

User and expert satisfaction with various FE models. (a) User satisfaction with various feature extraction models. (b) Expert satisfaction with various feature extraction models.