Transformer-based intelligent fault diagnosis methods of mechanical equipment: A survey

3.1.1 Encoder-decoder

Transformer employs an encoder-decoder model architecture that avoids loops, as shown in Figure 6. The first part is the encoder, which consists of six identical encoder layers stacked together. Each encoder layer consists of two sub-layers, namely, the multi-head self-attention (MSA) and the feed-forward neural network (FFN). MSA can focus on different positions in the input sequence, capturing global contextual information. FFN is used to perform nonlinear transformations on the features at each position. The entire encoder gradually extracts abstract representations of the input sequence through the stacking of multiple encoder layers.

Figure 6

The structure of encoder-decoder.

Next is the decoder. After the input data are processed by the encoder layers, they are passed to each layer of the decoder to compute attention scores. The decoder also consists of six identical decoder layers, each of which comprises three sub-layers: MSA, masked multi-head attention (MMSA), and FFN. MSA retrieves information about the input sequence from the output of encoders, helping the decoder generate the correct output sequence. MMSA is primarily employed to mask or hide information at certain positions to prevent the model from overly relying on previous tokens during generation, thus improving the model’s generalization ability and suppressing information leakage.

3.1.2 Feed-forward network

FFN is a fully connected feed-forward neural network that is added after the self-attention layer of each encoder and decoder. It receives the output of the self-attention layer as input and then outputs a new representation vector containing higher level semantic information. The computational procedure of FFN is as follows:

where W ₁ and W ₂ are the linear transformation matrices of the first and second fully connected layers, respectively, σ denotes the nonlinear activation function, and the dimension of the hidden layer is d _h = 2,048.

In FFN, a two-layer fully connected structure is used and a ReLU activation function is employed between the two layers. Specifically, in each FFN, the input representation vector is first linearly transformed through one fully connected layer, then nonlinearly transformed through a ReLU activation function, and finally linearly transformed through another fully connected layer to obtain the output. The advantage of FFN is that they can extract higher-level semantic features of the inputs through the nonlinear transformations of the multiple layers, which improves the expressive power of the models. In addition, since the computation of FFN is performed independently, the training process of the model can be accelerated by parallelization.

3.1.3 Position encoding

Since Transformer does not contain any recursive and convolutional structures to capture the positional information of different words in the text, some relative or absolute positional information about the tokens in the sequence has to be added in order for the model to be able to record the sequential relationships between the sequence data. To this end, position encoding is added to the bottom of the stack of the encoder and decoder, numbering the position of each word in the text, with each number corresponding to a word vector, respectively, and by combining the position vectors and the word vectors, certain positional information is added to each word. Compared with the sequential input method of RNN, Transformer can input data in parallel and retain the positional relationship between the data, which improves the computation speed and reduces the storage space requirement. Moreover, the dimensions of the position encoding and the input sequence embedding vectors are the same, so both can be summed up. Currently, there are various methods for position encoding [79], Transformer is realized by using sine and cosine functions with different frequencies [11], which can preserve the relative relationship of position information, and the specific operation can be expressed as follows:

where pos denotes the position of the word in the text each time, i represents the dimension, d _m is the dimension of the position encoding, 2i is the even dimension of the position encoding, and 2i + 1 represents the odd dimension of the position encoding ( i .e . 2 i ≤ d , 2 i + 1 ≤ d ) . It follows that each dimension of the position encoding corresponds to a sinusoidal wave with a geometric progression of wavelengths from 2π to 10,000−2π.

3.1.4 Self-attention

Attention mechanism [80] can be traced back to research in the field of neuroscience. Through the attention mechanism, the human brain is able to selectively focus on specific information and filter out irrelevant information when faced with a complex external environment. This mechanism allows us to better process and understand the information we receive. In the field of AI, the attention mechanism is introduced into DL models to improve the performance of the models. By introducing the attention mechanism, the model can dynamically assign different attention weights according to different parts of the input data and selectively process the important information. This mechanism can help the model better understand the input data and extract the key information in it. In recent years, the attention mechanism has been widely employed in speech recognition [81], machine translation [82], and image processing [83].

Self-attention [84] improves the attention mechanism in some details:

1. Objects of attention: the self-attention mechanism focuses on the correlation between different parts within the input information, while the attention mechanism mainly focuses on the correlation between the elements within the input utterance.

2. Information processing method: the information processing method of the self-attention mechanism does not rely on external information, but mainly on the characteristics of the input information itself. The attention mechanism, on the other hand, can utilize external information, such as contextual information, in its processing.

3. Performance: the self-attention mechanism can better capture the internal relevance of the input information, which is especially suitable for solving the long-distance dependency problem, whereas the traditional attention mechanism may encounter difficulties in dealing with this kind of problem.

By introducing the self-attention mechanism, Transformer completely abandons convolutional and recursive operations in its structure and relies only on the self-attention mechanism for global feature information extraction. The original Transformer proposes the scaled dot-product attention (SDPA) [11]. The basic structure of SDPA is depicted in Figure 7.

Figure 7

The basic structure of SDPA.

As is shown in Figure 7, Let Y ∈ ℝ n × d m be a sequence ( y 1 , y 2 , y 3 , … , y n ) containing n elements, where d m is the dimension of the element m. In the self-attention mechanism, three trainable weight matrices are defined as query matrix W Q ∈ ℝ n × d q , key matrix W K ∈ ℝ n × d k , and value matrix W V ∈ ℝ n × d v . Each element of the input sequence Y ∈ ℝ n × d m is linearly projected to each of the three weight matrices to generate three new vectors: the query vector (Query, Q ), the key vector (Key, K ), and the value vector (Value, V ), which are computed as follows:

SDPA, on the other hand, computes the dot product of the query vector Q and the key vector K with scaling, then performs Softmax normalization, and finally multiplies it with the value vector V to obtain the output matrix. The specific calculation formula is as follows:

where d k represents the dimensions of the vectors Q and K , and d k is the scale factor (square root of the dimension of the key vector K ).

Each word in the self-attention mechanism computes an attention score (weight) that reflects the word’s relevance to the other words in the sequence, thus capturing global information about the sequence and preserving information about long-term dependencies between different words.

MSA is an extension of self-attention and consists of multiple independent self-attention layers (heads), each of which has independently trainable weight matrices W i Q , W i K , and W i V . The basic structure of MSA is shown in Figure 8.

Figure 8

The basic structure of MSA.

The essence of MSA is to obtain multiple sets of queries, keys, and values by mapping the input sequences into different subspaces by linear transformations while ensuring that the number of parameters is overall constant. The specific calculation process is as follows:

where h is the number of heads of MSA, Z _i represents the output vector of each self-attention head, W ^P denotes the linear transformation matrix of the output vector of each self-attention head. Q _i , K _i , and V _i can be considered as multiple splits of Q , K , and V in the self-attention performed in different subspaces.

3.2.1 Vision Transformer

ViT is the first successful application of Transformer in the field of image classification, which exceeded the state-of-the-art CNN models ResNet [96] and EfficientNet [97] at that time in terms of image classification performance, and the model architecture is shown in Figure 9.

Figure 9

The model structure of ViT [75].

The key to applying ViT for image classification is to convert the image into sequential data. ViT segments the input image into a series of patches, each containing a portion of the image information. Each patch is then transformed into a vector representation, which is called embedding.

3.2.2 Data-efficient image Transformers

Based on knowledge distillation and self-supervised learning, DeiT needs less data and arithmetic power to achieve image classification results on the ImageNet dataset that are comparable to top CNN models. The model structure of DeiT is shown in Figure 10.

Figure 10

The model structure of DeiT [76].

As shown in Figure 10, the overall training process of DeiT consists of four stages: data preparation, model architecture, self-supervised pre-training, and supervised fine-tuning. Compared with the traditional ViT, DeiT can achieve better results with fewer samples or even no samples.

3.2.3 Swin Transformer

Traditional ViT models usually need to divide the whole image into multiple fixed-size image patches and then perform global attention computation. However, this approach tends to destroy the local information of the image. Swin Transformer introduces a hierarchical Transformer structure, as shown in Figure 11, which realizes multi-scale feature extraction by dividing the input image into a series of non-overlapping windows, and then the local self-attention computation within the windows is performed.

Figure 11

The hierarchical structure of Swin Transformer [77]. (a) Swin Transformer. (b) ViT.

As illustrated in Figure 11, the computational complexity of Swin Transformer varies linearly with the size of the input image, whereas the computational complexity of the ViT model varies as a square multiple of the size of the input image [98].

3.2.4 Tokens-to-token vision Transformer (T2T-ViT)

The core idea of T2T-ViT is to progressively refine image features through a recursive token transformation mechanism to improve the classification performance. The model structure of T2T-ViT is shown in Figure 12.

Figure 12

The model structure of T2T-ViT [87].

T2T-ViT improves the traditional ViT model by introducing a recursive token conversion mechanism and a deep narrow network structure, which improves the performance and efficiency of image classification.

3.2.5 Cross-attention multi-scale vision Transformer (CrossViT)

CrossViT improves the traditional ViT by combining multi-scale feature fusion and cross-attention mechanism. The model structure of CrossViT is shown in Figure 13.

Figure 13

The model structure of CrossViT [90].

As shown in Figure 13, CrossViT introduces a two-branch ViT structure, where each branch handles image features at different scales. This mechanism allows information to be exchanged between two branches, thus enabling feature fusion. CrossViT can make full use of both local and global information of an image to improve the classification performance.

3.2.6 Pyramid vision Transformer (PVT)

PVT combines the pyramid structure in CNN and the Transformer’s self-attention mechanism. The core idea of PVT is to construct a feature pyramid structure with different resolutions in the Transformer. The model structure of PVT is shown in Figure 14.

Figure 14

The model structure of PVT [86].

As demonstrated in Figure 14, PVT employs a down-sampling operation similar to that in CNN to gradually reduce the resolution of the feature maps, thus constructing feature pyramids with different resolutions.

3.3.1 Verification of fault diagnosis methods

Currently, the datasets used in the validation session of Transformer-based mechanical equipment fault diagnosis methods mainly include four kinds of public datasets, such as the Case Western Reserve University (CWRU) bearing dataset [99], XJTU-SY rolling bearing accelerated life test dataset [100], University of Connecticut (UCONN) gearbox fault dataset [101], and Harbin Institute of Technology (HIT) aero-engine inter-shaft bearing fault dataset [102], which contain a variety of bearing fault patterns, rolling bearing full-life cycle vibration data, multiple gearbox fault data, and the vibration signal of rotors and casings, respectively. The arrangement of the data acquisition equipment corresponding to the above four datasets is shown in Figure 15.

Figure 15

Data acquisition equipment: (a) CWRU dataset; (b) XJTU-SY dataset; (c) UCONN dataset; and (d) HIT dataset.

The CWRU bearing dataset contains a total of ten prefabricated faults of rolling bearings, which contain the normal condition as a special fault pattern, and has been widely applied in fault diagnosis research for rotating machinery. The XJTU-SY dataset is a widely employed rolling bearing accelerated life test dataset, which is collected and compiled by a team from Xi’an Jiaotong University in cooperation with Zhejiang Changxing Shengyang Science and Technology Co. Different from the CWRU bearing dataset, this dataset records the full life cycle vibration data of 15 rolling bearings under three operating conditions. The UCONN dataset is a gearbox dataset collected and arranged by the University of Connecticut from a two-stage gear box. The UCONN dataset contains 936 samples and 9 failure modes, which are normal condition, missing teeth, root fracture, contact surface spalling, and five kinds of tooth tip defects in different degrees. The HIT dataset, including three states of inter-shaft bearings, is proposed based on a real aero-engine by replacing the inter-shaft bearing with artificial fault, driven by motors and equipped with a lubricating system.

Through in-depth analysis, it can be concluded that the above public datasets reveal the following physical phenomena and physical principles:

1. Physical phenomena. In the normal operation of mechanical equipment, due to the periodic movement of rotating parts (such as bearings, gears, etc.), periodic vibration signals will be generated. These signals usually appear as distinct peaks on the spectrogram, related to the speed of the mechanical equipment and the natural frequencies of the components. When a component in the mechanical equipment fails (such as bearing damage, gear teeth broken, etc.), a transient impact component is generated in the vibration signal. These impact components often have a wide frequency band and high energy, which is an important basis for fault diagnosis. With the deterioration of mechanical equipment performance and the development of faults, the amplitude and frequency components of vibration signals tend to change. For example, the amplitude may gradually increase, while certain frequency components may disappear or new frequency components may appear.

2. Physical principles. When the vibration frequency of a mechanical device is close to the natural frequency of a component, a resonance phenomenon will occur, resulting in a significant increase in vibration amplitude. This phenomenon is of great significance in fault diagnosis because the resonance frequency is often related to the type of fault. In the vibration process of mechanical equipment, energy is transferred and converted between the various components. When a component fails, it will lead to abnormal energy transmission and conversion, which reflects the fault information in the vibration signal. The vibration signal of mechanical equipment can be regarded as the propagation of mechanical waves in the medium. The wave principle can be used to explain the attenuation, scattering, and interference of vibration signals in the process of propagation, and provide theoretical support for fault diagnosis.

The performance of different methods on these four public datasets are summarized in Table 4.

Analysis of Table 4 shows that the fault diagnosis accuracy of the HIT dataset is significantly lower than that of the other three datasets, which is due to the fact that the HIT dataset is closer to the actual fault diagnosis situation of mechanical equipment, which is more challenging compared to the dataset obtained under laboratory conditions, and the dataset provides a new benchmark for the validation of mechanical equipment fault diagnosis methods.

3.3.2 Related literature review of Transformer-based intelligent fault diagnosis methods

Driven by the first Transformer-based intelligent fault diagnosis research idea of mechanical equipment, Jin et al. [106] proposed a fault diagnosis method for rotating machinery based on time series Transformer (TST) to solve the problem of long-term dependence of traditional CNN and RNN-based fault diagnosis models. The overall model architecture of TST is shown in Figure 16.

Figure 16

The overall model architecture of TST [106].

As shown in Figure 16, a new time series tagger is first designed for one-dimensional data processing, and then TST is proposed on this basis in conjunction with the Transformer model. Finally, the CWRU dataset, XJTU-SY dataset, and UCONN dataset were used to verify the validity of the model. Experimental results show that TST achieves 98.63% (ten categories), 99.72% (four categories), 99.78%, and 99.51% fault diagnosis accuracy for the above three datasets, respectively, which are higher than the traditional CNN and RNN models. Meanwhile, the results after feature visualization using t-SNE also proved that the feature vectors extracted by TST have the best intra-class closeness and inter-class separability, further proving the effectiveness of the method.

Hou et al. [109] proposed a bearing fault diagnosis method based on joint feature extraction of Transformer and ResNet (TAR) for the problems of difficult data acquisition, unbalanced category distribution, and noise interference that often exist in DL models for bearing fault diagnosis driven by big data. The overall model architecture of TAR is shown in Figure 17.

Figure 17

The overall model architecture of TAR [109].

As indicated in Figure 17, feature separation and word embedding were first performed on the original one-dimensional signals through a one-dimensional convolutional layer, which were transmitted to the Transformer encoder and ResNet framework for feature extraction, respectively, and the diagnostic accuracy was better than that of the traditional DL network. In addition, the migration learning strategy employing model fine-tuning reduced the training difficulty of the method in new tasks. Finally, the CWRU dataset was used to verify the validity of the model. The experimental results show that TAR achieves up to 99.90% fault diagnosis accuracy for the CWRU dataset without adding noise, and when adding noise with different signal-to-noise ratios, TAR’s average fault diagnosis accuracy is higher than that of the comparison methods.

Fang et al. [115] explored a lightweight Transformer based on convolutional embedding and linear self-attention, named CLFormer, for fault diagnosis of rotating machinery. The overall model architecture of CLFormer is shown in Figure 18.

Figure 18

The overall model architecture of CLFormer [115].

To begin with, the input original one-dimensional signal was normalized in [−1,1], the convolutional embedding module is constructed to replace the original embedding module, to reduce the complexity of the model, the linear self-attention was used to replace the original self-attention, which makes the CLFormer satisfy the demand of lightweight. Finally, the effectiveness of the proposed method was evaluated on a laboratory-measured rotating machinery dataset. The experimental results show that compared with Transformer, the number of parameters of CLFormer decreases from 35.22 to 4.88 K, and the accuracy of fault diagnosis increases from 82.68 to 90.53%, which is of practical application value.

Aiming at the problems of low accuracy and poor robustness of traditional rolling bearing fault diagnosis based on DL, Hou et al. [107] designed a multi-feature parallel fusion rolling bearing fault diagnosis method with Transformer as the basic network, named Diagnosisformer. The overall model architecture of Diagnosisformer is shown in Figure 19.

Figure 19

The overall model architecture of Diagnosisformer [107].

The fast Fourier transform is primarily used to extract the frequency-domain features of the original one-dimensional vibration data, and then the model inputs are subjected to normalization operations and embedded into the network. Then, the multi-feature parallel fusion encoder is used to extract the local and global features of the bearing data, and the corresponding features are passed to the cross-flipped decoder and classified by the classification head for fault classification. Finally, self-made rotating machinery fault diagnosis data and the CWRU dataset were used to verify the validity of the model. The experimental results show that Diagnosisformer achieves an average diagnosis accuracy of 99.84 and 99.85% for the above two datasets, respectively. Accuracy and robustness are significantly better than CNN, CNN-LSTM, RNN, LSTM, GRU, and other methods.

Yang et al. [108] proposed a signaling Transformer (SiT) on the basis of the attention mechanism and applied it to a study of bearing fault diagnosis. The overall model architecture of SiT is shown in Figure 20.

Figure 20

The overall model architecture of SiT [108].

As demonstrated in Figure 20, the original one-dimensional vibration time series was first segmented, then the segmented subsequence was linearly encoded and positionally encoded, and finally the encoded subsequence was input into the Transformer for feature extraction to realize the fault pattern recognition of bearings. The effectiveness of the method was finally verified using the CWRU bearing dataset and the self-made centrifugal pump bearing dataset. The experimental results show that SiT achieves an average diagnostic accuracy of 99.46 and 99.53% for the above two datasets, respectively, which verifies the effectiveness of the proposed method.

Aiming at the problem that the self-attention mechanism in the current Transformer model can only focus on the correlation information within the sequence and cannot understand the information gap between the samples, Li et al. [116] proposed a Twins Transformer model based on two-branch Twins attention for bearing fault diagnosis. The overall model architecture of Twins Transformer is shown in Figure 21.

Figure 21

The overall model architecture of Twins Transformer [116].

The proposed Twins Transformer uses cross-attention for the first time to compute correlation information between samples. In addition to retaining the correlation information within the sequence data obtained by computing the self-attention, the cross attention is also utilized to learn the correlation information between the samples. Finally, the performance of the model is validated on four commonly used bearing datasets, the CWRU dataset, UPB dataset [117], MFPT dataset [118], and JUN dataset [119]. The average accuracy of each dataset is improved by 1.73–99.42% compared to the original Transformer.

In terms of visual Transformer applications, Ding et al. [120] processed the original one-dimensional vibration signal of rolling bearings via synchronous compression WT to obtain a multi-channel time-frequency representation, which was then input into a new time-frequency Transformer (TFT) to extract discriminative hidden features and accurately classify fault types. The overall model architecture of TFT is shown in Figure 22.

Figure 22

The overall model architecture of TFT [120].

Finally, a self-made bearing experiment dataset was used to verify the validity of the model. Compared with other DL models, this method had higher diagnostic accuracy and faster training speed. Thus, the superiority of the method is demonstrated.

Tang et al. [121] explored an integrated ViT model based on WT and soft voting for bearing fault diagnosis. The overall model architecture of the integrated ViT is shown in Figure 23.

Figure 23

The overall model architecture of the integrated ViT [121].

First, the original one-dimensional vibration signal was decomposed into sub signals of different frequency bands via discrete wavelet transform, and then these sub signals were transformed into wavelet time-frequency maps using CWT. Second, the wavelet time-frequency maps were input into ViT for preliminary diagnostic analysis. Finally, the soft voting method was employed to fuse all preliminary diagnostic results and obtain the final diagnostic decision result. The CWRU dataset was used to verify the validity of the model. The experimental results show that the integrated ViT has the highest fault diagnosis accuracy on all the three datasets, which are 100, 99.67, and 99.83%, respectively, which is better than the integrated CNN and ViT and other comparison methods.

Fan et al. [122] proposed a ViT-based fault diagnosis method of rolling bearings to improve the accuracy of rolling bearing fault diagnosis. The overall ViT model architecture is shown in Figure 24.

Figure 24

The overall ViT model architecture [122].

As illustrated in Figure 24, the original one-dimensional vibration signal was transformed into a grayscale texture image through local binarization, segmented into predetermined sized small blocks, and then transformed into a sequence through linear mapping. The global information of the image was extracted through self-attention mechanism to achieve bearing fault diagnosis. To improve the image recognition performance of ViT, pooling layers were introduced, and the accuracy of the new pooling ViT was improved by 3.3% compared to the original ViT.

Cui et al. [123] conducted research on fault diagnosis of waterborne diesel engines under various internal and external excitations, and a fault diagnosis method based on the complementary ensemble EMD of adaptive noise, signal-to-image conversion, and Swin Transformer was proposed. The overall Swin Transformer model architecture is shown in Figure 25.

Figure 25

The overall Swin Transformer model architecture [123].

As is shown in Figure 25, the original one-dimensional vibration signals were first decomposed to obtain a time-frequency matrix. Second, the time-frequency matrix was converted into a two-dimensional color image through signal-to-image conversion and pseudocolor encoding. Finally, the two-dimensional image was input into Swin Transformer to fully extract feature information for fault diagnosis. The self-made marine diesel engine dataset was used to verify the effectiveness of the method, and the experimental results proved that the method can effectively extract the fault feature information, and the average fault diagnosis accuracy can reach 98.3%, and has a certain degree of anti-noise interference ability.

Jin et al. [124], aiming at the reality that traditional bearing fault diagnosis methods rely on a large amount of manually labeled data, proposed a bearing intelligent fault diagnosis method based on WT and self-supervised learning, which effectively solves the problem of insufficient fault training samples. The overall model architecture is shown in Figure 26.

Figure 26

The overall model architecture [124].

First, WT and cubic spline difference methods are used to convert the original one-dimensional vibration data into two-dimensional wavelet time-frequency maps, which are then input into the ViT network for feature extraction, KNN is used for fault classification, while the label-free self-distillation algorithm is utilized to solve the problem of self-supervised learning for both finite-labeled data and sufficiently unlabeled data. Finally, the CWRU dataset and XJTU dataset are employed to validate the model. The experimental results show that the method can obtain more than 90% average fault diagnosis accuracy in both datasets with only 1% labeled data, and the comparison results with other self-supervised learning methods also prove the effectiveness and superiority of the method.

He et al. [110] proposed a bearing fault diagnosis method Siamese vision Transformer by fusing Siamese network and ViT under the conditions of limited labeled training data and complex working conditions, and the overall model architecture is shown in Figure 27.

Figure 27

The overall model architecture of Siamese vision Transformer [110].

Firstly, the STFT is used to convert the one-dimensional vibration signal of the bearing into a two-dimensional time-frequency map, and the feature vectors of the input samples are efficiently extracted in the high-order space to accomplish the fault diagnosis. In the training process of the model, the loss function combining the KL dispersion in both directions and a new random masking strategy are proposed and designed. Finally, the CWRU dataset and Paderborn dataset are used to validate the model. The experimental results show that the proposed method achieves 97.56 and 98.11% average fault diagnosis accuracies, respectively, with limited data, demonstrating the generalization and effectiveness of the method.

3.3.3 Summary of research status

The current research status demonstrates that although some achievements have been made in the research of Transformer-based intelligent fault diagnosis of mechanical equipment, Transformer has attracted the attention of some scholars due to its advantages such as self-attention mechanism, parallel computing, multi-task learning, and flexible and scalable structure. Transformer has gained rapid and wide dissemination in the 6 years since it was proposed, and has made impressive achievements in many tasks such as NLP, speech recognition, image processing, recommender systems, etc. However, so far, the research and applications related to mechanical equipment fault diagnosis using Transformer and its variants are still very limited and have the following problems:

1. The structure of Transformer is relatively complex and contains a large number of parameters, which requires large-scale training samples when pre-training the model and consumes a large number of computational resources and time, increasing the risk of overfitting. For some small datasets or domain-specific fault diagnosis tasks, Transformer may not be able to fully learn the fault features, which leads to poor fault diagnosis results.

2. Currently, most of the Transformer-based mechanical equipment fault diagnosis studies collect mechanical equipment state information under stable operating conditions with fixed operating parameters, environmental variables, etc., and many of the studies only analyze single fault patterns. However, the state of mechanical equipment in the actual operation process is mostly a complex fault pattern that occurs across working conditions such as compound speed or multiple faults. Therefore, only the study of stable working conditions or a single fault pattern for fault diagnosis of mechanical equipment does not meet the actual situation.

3. The application of Transformer in the field of mechanical equipment intelligent fault diagnosis is common in rolling bearings, rotor systems, and other common rotating machinery, and there is a great lack of research on typical reciprocating machinery such as diesel engines, and the method validation is mostly based on the public dataset, and there are fewer studies based on the data collected from actual industrial equipment.

4. Research on intelligent fault diagnosis of mechanical equipment based on Transformer has achieved a series of remarkable results. These results not only validate the effectiveness and accuracy of Transformer methods, but also reveal the close relationship between relevant findings and physical phenomena. For example, some studies have found that certain vibration signals of specific frequencies are highly correlated with certain types of mechanical equipment faults. This discovery not only provides a new characteristic index for fault diagnosis, but also provides valuable reference information for the maintenance and management of mechanical equipment. In addition, it has been found that Transformer models can automatically learn some feature representations related to physical phenomena when processing vibration signals. These feature representations not only help to improve the accuracy of fault diagnosis, but also provide a new perspective for understanding the operation mechanism and fault process of mechanical equipment.