Fault diagnosis of agricultural sprinkler irrigation machinery equipment based on machine vision

3.1 FD based on MV technology

ASIME is a device that sprays water with a certain pressure into the air through specialized machinery and equipment, scattering it into small water droplets that evenly fall on the field like a drizzle, providing water for crops, flowers, seedlings, and other plants. Connecting devices such as pressurization, water delivery, spraying, and walking into a movable whole is called sprinkler irrigation machinery. Sprinkler irrigation systems are machinery and equipment used to spray irrigation water resources in fields. They come in many forms, and their main accessories include supporting power machines, pipeline systems, nozzles, and other components [19]. There are many commonly used types of sprinkler irrigation machines, which can be divided into rolling type, translation type, etc. The specific classification is shown in Figure 1.

Figure 1

Classification of sprinkler.

In Figure 1, among all types of sprinkler irrigation machines, light sprinkler irrigation machines mainly consist of nozzles, spray racks, outlet pipes, power machines, sprinkler pumps, inlet pipes, filters, racks, etc. The rotating bearings in components such as power machines, water pumps, and sprinklers have the highest probability of failure due to factors such as temperature and acid-base substances. Therefore, research will focus on diagnosing the faults of these components in sprinkler irrigation machines. Rotary power in sprinkler systems comes primarily from water pressure and secondarily from the design of the sprinkler head. That is, sprinkler heads in sprinkler systems are usually driven to rotate water jets by water flow pressure. Therefore, in the design of the sprinkler system, the water supply pressure of the water source is considered to ensure that the sprinkler head can obtain enough water pressure to achieve rotation. In the study, the water pressure is controlled to 0.6 MPa and the sprinkler head can be designed as a rotating jet, fan jet, or spray jet and other forms in order to produce a rotating effect through the power of the water flow. MV technology is an interdisciplinary field that involves artificial intelligence, image processing, and many other fields. MV mainly utilizes computers to imitate human visual functions, extracts information from images, processes and understands it, and ultimately, applies it for practical detection, measurement, and control. This technology has the characteristics of fast speed and large amount of information, so the research will use it to detect faults in ASIME. Due to the fact that mechanical faults are often accompanied by vibration changes, when using MV technology for FD, the type of fault can be determined by judging the changes in mechanical vibration signals. The collection of mechanical vibration signal images can be achieved using a visual acquisition system, which is composed of high-speed industrial cameras, non-stroboscopic light sources, multi-channel transmission cables, etc. The ordinary camera in a vision acquisition system is affected by pixel resolution, frame rate, exposure time, and light source conditions during the process of image acquisition. Therefore, the camera used in the study is a high-speed industrial camera and provides a non-frequency flash source. High-speed industrial camera has the advantages of high resolution, high-speed shooting, good stability, programmable control, fast data transmission and suitable for automated production lines, which can reduce the impact of the abovementioned influencing factors on image acquisition. Based on this, the research for collecting images of mechanical vibration signals can use the visual acquisition system, which is mainly composed of a high-speed industrial camera, a strobe-free flash source, a multi-channel transmission cable and so on. The workflow is shown in Figure 2 [20].

Figure 2

Workflow of the visual acquisition system.

In Figure 2, when the visual acquisition system is working, the software is connected to the hardware device, and then the acquisition card is connected and initialized. Next the high-speed camera is connected and the camera parameters are initialized. After connecting various components, it is necessary to check if the connection is successful. If it is not successful, it needs to be reconnected. The image captured by the camera is checked and adjusted, relevant parameters are set, and finally the image is captured and saved. When the image meets the shooting requirements, the collection work is finished. If it does not satisfy the demands, it needs to be retaken. After collecting mechanical vibration images using a visual acquisition system, a phase-based motion extraction method is utilized to extract the vibration acceleration signal. The calculation method is shown in formulas (1)–(4).

where L d ( x , y , t 0 ) means the local amplitude of the image; p d ( x , y , t 0 ) means the local phase of the image; S ( x , y , t 0 ) denotes the image intensity; T d + i X d expresses the complex form of Gabor filters; i indicates the imaginary unit; t 0 refers to the time. x and y both stand for spatial positions.

where w represents the sine wave wavelength of the filter; d means the parallel stripe direction of the filter core; ψ refers to the phase offset; γ denotes the spatial length ratio; σ is the standard deviation of the Gaussian function.

where x d and y d represent the spatial displacement vectors of the object in the horizontal and vertical directions, respectively.

where x v and y v indicate the velocities in the horizontal and vertical directions, respectively; ∂ p 0 ( x , y , t ) ∂ x expresses the differentiation of local phase P d ( x , y , t ) to x in the horizontal direction; ∂ p 0 ( x , y , t ) ∂ t represents the differentiation of local phase P d ( x , y , t ) over t in the horizontal direction; ∂ p π/2 ( x , y , t ) ∂ y refers to the vertical differentiation of local phase P d ( x , y , t ) over y ; ∂ p π / 2 ( x , y , t ) ∂ t indicates the vertical differentiation of local phase P d ( x , y , t ) over t . Based on the vibration signal information, the time domain information of the vibration signal can be obtained by performing fast Fourier transform (FFT), and the resulting spectrum chart records the coordinates of the peak points in the spectrum chart from left to right, as shown in Figure 3.

Figure 3

Spectrum diagram of the vibration signal. (a) Vibration time-domain signal. (b) Vibration frequency-domain signal.

In Figure 3, the vibration signal is converted into frequency-domain information, and the peak coordinate information in the frequency-domain is recorded to form a vector matrix, as shown in formula (5).

where the frequency-domain peak coordinate information is recorded by S 1 ; S 2 ; S 3 ; … ; S N under different speed conditions; P 1 stands for the fault sample data of a certain type of SI machinery. By applying the same processing to other types of faulty machinery, frequency-domain data of different types of mechanical faults can be obtained, as shown in formula (6).

where P 1 ; P 2 ; P 3 ; … ; P N represent the frequency-domain data of N types of sprinkler irrigation machinery under different rotational speed conditions, and ASIME diagnosis is conducted based on these data.

3.2 FD of agricultural machinery design based on joint deep learning and MV

When using MV technology for FD of ASIME, mechanical photos are converted into frequency-domain data for diagnosis. However, during the diagnosis, it is found that the processing of a large amount of fault data is very complex and prone to diagnostic errors. In the FD of ASIME, the MV system can be used to recognize whether there is a fault in the equipment by monitoring the motion state, vibration, sound, and other information. Among them, the momentum principle can be used to analyze the kinetic characteristics of the equipment in the process of movement, and then determine whether there is a fault in the equipment. For example, when the equipment operates normally, its motion state should comply with the law of conservation of momentum, i.e., the sum of external and internal forces on the equipment is zero. When the equipment is faulty, due to the problem of faulty components, it may lead to changes in the mechanical properties of the equipment in the process of movement, making the law of conservation of momentum no longer hold. By analyzing the kinetic characteristics of the equipment in the process of movement, the type of fault that exists in the equipment can be identified, and then carry out the corresponding maintenance and treatment. CNN is a mathematical model simulating biological neural network, which has great advantages in data processing [21,22]. Therefore, the study combines it with MV technology for FD of ASIME. CNN mainly includes convolution, excitation, pooling, and fully connected layers. The structural diagram is shown in Figure 4.

Figure 4

Schematic representation of the CNN structure.

Convolutional operation is the core of CNN algorithm, and its main function is to filter the input signal, extract feature information, and then transmit it to the pooling layer to form a feature map. In the network structure of CNN, there are a large number of convolutional layers, and the latter layer reprocesses the input data of the previous layer to obtain deeper features. The convolution calculation is shown in formula (7).

where x k n means the output of the k th convolutional kernel after n layer convolution; x i n − 1 indicates the input of the n convolutional layer and the output of the n − 1 layer; M k denotes a selected input feature mapping; ω i k n is the convolutional kernel weight; b k n denotes offset. The excitation layer, also known as the activation function layer, nonlinearly transforms the input signal to increase the expressive and generalization capabilities of the model. It also compresses and stretches the input signal to reduce the effect of noise and disturbance on the model. The excitation layer in the FD model of ASIME has the functions of nonlinear transformation, increasing model robustness, sparsity constraints, and feature mapping, which help to improve the model’s FD accuracy and generalization ability. The pooling layer is located after the excitation layer. Maximum pooling has a strong ability to extract periodic signal features and is the most widely used method in CNN. The calculation method is expressed in formula (8).

where p k n + 1 refers to the output value of the kth channel in the n + 1 layer; S means the width of the pooling window; j indicates the j th moving step; q k n ( t ) expresses the k th channel in the q k n ( t ) layer; t denotes the value of neurons. Finally, the input of the pooling layer is used as the fully connected layer’s input, and high-dimensional features are obtained through convolution, excitation, and pooling methods. Low-dimensional features are obtained through Flatten function expansion, and output to Softmax. The fully connected calculation formula is shown in formula (9).

where B k j + 1 means the output of the k th neuron in the j + 1 layer of the fully connected layer; ω i k j indicates the weight of the JJ neuron in the j layer of the fully connected layer to the i neuron in the j + 1 layer; b k j is the bias of all neurons in the j layer of the fully connected layer toward the k neuron in the j + 1 layer of the fully connected layer. However, traditional CNN has high requirements for data quality, and when there are many types of sample data and high complexity, its accuracy will be greatly reduced. In the FD of ASIME, the large amount of data samples and the complex environment can affect the accuracy of CNN. The SAE algorithm is formed by stacking multiple autoencoders (AE) together. SAE, which can effectively extract features from data and meet requirements by reconstructing data, has been broadly utilized in fields like image recognition and video analysis [23]. Therefore, the study will use SAE algorithm to compensate for the shortcomings of CNN algorithm in FD of sprinkler machinery equipment. The AE algorithm takes the input signal as the learning object and achieves data denoising and visual dimensionality reduction through feature learning. This algorithm divides the self-compiled program into two modules. The structure of the automatic encoder is shown in Figure 5.

Figure 5

Schematic diagram of AE structure.

In Figure 5, the entire AE process can also be represented mathematically, as shown in formulas (10) and (11).

where f means the activation function from the input layer to the hidden layer; ω 1 denotes the weight; b 1 means the deviation; X indicates the input value; Y expresses the hidden layer’s output value.

where g is the activation function from the hidden layer to the output layer; Z is the output value. In the encoding and decoding process, the ideal state is to make the input and output values the same. To reduce the loss value of input data and output data, in addition to using the same calculation method for input and output, a loss function needs to be introduced. The commonly used loss function of AE is the mean square error function, and the function expression is shown in formula (12).

where n 1 expresses the total amount of input data. In practical applications, stacking multiple AEs together to form a deeper neural network called SAE can make up for the weaknesses of a single hidden layer in extracting data features. Therefore, the SAE algorithm can be used to improve the CNN algorithm, making up for the shortcomings of CNN in FD of sprinkler machinery equipment, thereby improving the accuracy and accuracy of sprinkler machinery equipment FD. SAE-CNN is used to process the frequency-domain dataset constructed using MV technology, convert it into a grayscale image, partition the dataset according to a certain proportion, and perform FD. The workflow is shown in Figure 6.

Figure 6

FD process of combined deep learning and MV.

In Figure 6, high-speed cameras are first used to obtain vibration images of different fault categories, and then the frequency-domain signals of vibration displacement are extracted using the video phase extraction method. Grayscale image transformation is performed on the original frequency-domain displacement signal obtained, and the grayscale image data are broken into training, validation, and test sets grounded on a certain proportion. The reliability of the network is trained and verified using the divided data training and validation sets, and data features are extracted using SAE, and then the test dataset is used to obtain FD results for sprinkler machinery.

4.1 Comparison and analysis of SAE-CNN algorithm performance

To evidence the superiority of the SAE-CNN algorithm proposed by the research institute in FD, this study conducted performance comparison experiments among the SAE-CNN and Support vector machine (SVM), CNN, and SAE algorithms. The experimental environment for the comparative test is shown in Table 1.

In the test environment shown in Table 1, Iris dataset from the University of California (UCI) was also selected as the test set for this comparative test, and the accuracy, recall, precision, Receiver operating characteristic (ROC) and other indicators of the four algorithms were taken as the test indicators. The Iris dataset is a frequently utilized classification experiment dataset, called the Iris Flower Dataset. It contains 150 data samples, which are divided into 3 categories, with 50 data in each category and 4 attributes in each category. The accuracy and ROC results of the four algorithms are indicated in Figure 7.

Figure 7

Accuracy of four algorithms and ROC results. (a) Accuracy. (b) ROC curve.

Figure 7(a) shows that the SAE-CNN algorithm’s accuracy after stable training was significantly higher than other algorithms, with the accuracy of 95.13, 81.23, 78.63, and 70.21% for SAE-CNN, SVM, SAE, and CNN algorithms, respectively. Figure 7(b) shows that the area under the ROC of SAE-CNN, SVM, SAE, and CNN algorithms was 0.82, 0.70, 0.68, and 0.64, respectively. The above outcomes denoted that the effectiveness of SAE-CNN algorithm was better than the comparison algorithm from the perspective of accuracy and ROC. This was because the proposed SAE-CNN algorithm introduced a loss function, namely, the mean square error function, which effectively reduced the loss value of the input data and the output data, and further improved the performance of the algorithm. The four algorithms’ training errors and accuracy results are indicated in Figure 8.

Figure 8

Training error and training accuracy results of four algorithms. (a) Training error. (b) Training accuracy.

Figure 8(a) shows that after stable training, the training error of the SAE-CNN algorithm was significantly lower than that of other algorithms. Among them, the training error of the SAE-CNN algorithm was 0.012%, which was lower than 0.058% of the SVM algorithm, 0.064% of the SAE algorithm, and 0.071% of the CNN algorithm. The analysis showed that the training error of the proposed SAE-CNN algorithm was extremely low, which may be because SAE algorithm further realizes data noise reduction and visual dimension reduction through feature learning, and better helps the feature learning process. At the same time, this was closely related to the training process being long enough to learn more details and noise on the training data. Figure 8(b) expresses that the training accuracy of SAE-CNN, SVM, SAE, and CNN algorithms was 0.95, 0.83, 0.81, and 0.62, respectively. The above outcomes indicated that from the dimensions of training error and training accuracy, the performance of SAE-CNN algorithm was also superior to the comparison algorithm. In addition, the recall and accuracy of the four algorithms were organized to obtain the recall and Precision–Recall (PR) curve results shown in Figure 9.

Figure 9

Comparison results of recall rates and PR curves of four algorithms. (a) Recall. (b) PR curve.

Figure 9(a) shows that the recall rate of SAE-CNN algorithm was significantly higher than other algorithms, with the recall rates of SAE-CNN, SVM, SAE, and CNN algorithms being 89.7, 81.3, 79.8, and 76.1%, respectively. Figure 9(b) shows that the areas under the PR curve of SAE-CNN, SVM, SAE, and CNN algorithms were 0.87, 0.82, 0.79, and 0.61, respectively. The above findings indicated that from the dimensions of recall rate and PR curve, the performance of SAE-CNN algorithm was also better than that of the comparison algorithm. The precision of four algorithms was tested in the end of the study, and the results are shown in Figure 10.

Figure 10

Precision values of four algorithms.

According to Figure 10, the precision of the SAE-CNN algorithm is as high as 95.16% on average, while the precision of SVM, SAE, and CNN algorithms are 86.15, 82.33, and 76.81%, respectively, all lower than the SAE-CNN algorithm. The performance of SAE-CNN algorithm is significantly better than other algorithms. Based on the comparative analysis of the above dimensions, the SAE-CNN algorithm proposed in the study had superior predictive performance. Therefore, this study will apply it to the FD of sprinkler machinery, hoping to promote the development of the field of sprinkler machinery FD.

4.2 Performance comparison and analysis of fault detection models based on SAE-CNN algorithm

To analyze the fault detection effect of the SAE-CNN fault detection model, this experiment selected the mechanical failure prevention technology (MFPT) dataset to test it and similar CNN and SVM fault detection models. The MFPT dataset is a publicly available dataset for mechanical FD and prediction. This dataset was explored by the National Institute of Standards and Technology in the United States to provide a standardized testing platform for evaluating and comparing the performance of different FD and prediction algorithms. The MFPT dataset contains various types of mechanical faults, such as bearing faults, gear faults, motor faults, etc. Each type of fault has multiple instances, covering faults of varying degrees and frequencies. To compare the FD performance of the three algorithms in MFPT dataset, the MFPT dataset was divided into a training set (70%) and a test set (30%). SVM was used for classification, while CNN constructed standard CNN for classification. Then, the training set was trained and the model parameters were optimized by cross-validation. Finally, the performance of each model was evaluated on the test set. The results obtained from the testing of three different algorithms are expressed in Table 2.

From Table 2, among the 100 randomly chosen test samples in the MFPT dataset, the SAE-CNN fault detection model achieved an accuracy of 90.12% in 30 tests, while the SVM and CNN fault detection models achieved 72.35 and 66.74%, respectively. In 60 tests and 90 tests, the SAE-CNN fault detection model’s accuracy was 94.56 and 98.45%, respectively, which were higher than those of the comparative fault detection model. In addition, the calculation time of the proposed model in 30, 60, and 90 tests was 9.892, 8.926, and 6.863 s, respectively, showing an obvious decreasing trend. Compared with the other two models, the calculation time of the proposed model was obviously less, and the efficiency was higher. From the above results, the fault detection model proposed in the study had better fault detection effectiveness than the comparative model. Besides, to further analyze the performance of the SAE-CNN fault detection model proposed in the study, the prediction error comparison results of three FD models in the MFPT dataset and the scoring results of relevant practitioners were collected and organized. The results are shown in Figure 11.

Figure 11

The prediction error of the SAE-CNN FD model. (a) Comparison of prediction errors of three algorithms. (b) Scoring results of three fault prediction models.

Figure 11(a) shows that the prediction error of the SAE-CNN FD model fluctuated around 0, and its maximum positive and negative errors in 40 samples were 0.012 and −0.003, respectively, which were superior to the SVM and CNN FD models. Figure 11(b) shows that five groups of relevant practitioners rated the SAE-CNN FD model higher than the other two FD models. The average score of the SAE-CNN FD model was 8.9 points, higher than the 7.5 and 6.7 points of the SVM and CNN FD models, respectively. The SAE-CNN FD model was analyzed and found to have high requirements on the amount of data, the coverage of fault samples, and the model’s environment, which may also be the reason for its FD score of 8.9. The reason is that by stacking multiple AE together to form a deeper neural network (SAE), this study can make up for the deficiencies of a single hidden layer in extracting data features, and further improve the accuracy of FD of sprinkler irrigation machinery and equipment. Therefore, in specific FD, the model needs to be provided with a complete set of data and mature environmental conditions. Although the FD score did not reach more than 9, the predictive performance of the SAE-CNN FD model proposed in the study was better compared with the existing FD models and has been recognized by practitioners. Therefore, its application to the FD of actual sprinkler irrigation machinery can effectively improve the accuracy of FD, thus promoting the development of the field of sprinkler irrigation machinery.