Research on maintenance spare parts requirement prediction based on LSTM recurrent neural network

2.1 Net structure

LSTM is a chain-like neural network with repeated network modules. As shown in Figure 1, each module includes a three-layer network structure, including an input layer, a hidden layer, and an output layer. Each layer of the network is composed of a different number of parallel neural units. The input layer is responsible for receiving information. The component data in the current time data information X ( t ) , the module state information C ( t − 1 ) at the previous time, and the module output information H ( t − 1 ) are input into the neural units of the input layer according to different weights. The hidden layer is responsible for calculating and processing the input data information. The output of the output layer includes module status information C ( t ) , module output H ( t ) , and timing output y ˆ ( t ) . Each network module carries on data transmission through C ( t ) and H ( t ) , and timing output y ˆ ( t ) is the calculation output of the network at the current moment.

Figure 1

Logical structure diagram of LSTM recurrent neural network.

2.2 Neural unit design and memory realization

The core idea of LSTM is the design of adding unit state c ( t ) and “gate” structure to the hidden layer neural unit. Figure 2 shows the structure of one neural unit in the hidden layer of the network module at time t . The unit state c ( t ) serves as the link of information transmission, allowing the information of each time sequence to be transmitted between network modules at successive moments in the entire network, which is equivalent to long-term memory. Through the state information, the information from the earlier time can be transferred to the later time, which overcomes the influence of short-term memory. Forgetting gates, input gates, and output gates are mainly used for the calculation and processing of data information at the current time and state information at the past time, and are a comprehensive application of short-term memory and long-term memory.

Figure 2

LSTM recurrent neural network hidden layer neural unit structure diagram.

3.1 Algorithm procedure

According to the characteristics of the time series data with equipment spare parts consumption, this article proposes to use LSTM recurrent neural network to predict the requirement of equipment spare parts, in which the technical advantages of LSTM recurrent network will be fully performed. The detailed design of network structure, and the concrete realization of network training and network prediction are shown.

In order to solve the problem that the LSTM recurrent neural network prediction model involves many parameters, which will make the performance of prediction unstable if set artificially, with the goal of minimizing the prediction error, an algorithm of equipment spare part requirement prediction with LSTM recurrent neural network whose parameters are optimized by particle swarm optimization is further proposed. The structural framework of the algorithm is shown in Figure 3.

Figure 3

PSO optimizes the LSTM to predict the spare parts requirement.

The specific steps are as follows:

Step 1: Standardize the experimental data and divide it into a training set and a testing set.

Step 2: Set particle swarm algorithm parameter learning factors c 1 and c 2 , maximum inertia weight w max , minimum w min , number of iterations N iter , and number of populations p o p _ s i z e . Setting parameters such as the number of neural units in hidden layer of LSTM recurrent neural network s i z e _ h , prediction interval T , the number of training data subsets MiniBatchSize, and the initial learning rate η as state vectors, initialize the state and velocity of each particle in the population.

Step 3: To each particle, construct an LSTM recurrent neural network according to its four-state vector values, use the training set data to train the network, use the testing set data to test the network, and use the mean square error as the fitness value of each particle.

Step 4: Determine the extreme value p best of the individual and the extreme value g best of the population according to the fitness value of each particle in the population.

Step 5: According to p best and g best , update the state and velocity of each particle in the population.

Step 6: Calculate the fitness value of each particle in the population, and update p best and g best .

Step 7: Repeat steps 4 to 6, and after reaching the number of iterations, use the four-state vectors of particle g best as parameters to construct an LSTM recurrent neural network to predict the requirement of maintenance spare parts.

3.2 Data processing and network construction

The spare parts consumption data is the time series data P = { p 1 , p 2 , ⋯ p t , ⋯ , p n } with monthly intervals, p t represents the consumption of spare parts in tth month. p t = { k 1 t , k 2 t , ⋯ k m t } and k m t represent the consumption quantity of the mth kind of spare parts in t-month. In order to accelerate the convergence of weight parameters when training the network, first use method of z-score to normalize p t . The calculation formula for standardization of time series data is:

The standardized data set P ′ is divided into training set P t r ′ = { p 1 ′ , p 2 ′ , ⋯ , p x ′ } and testing set P t e ′ = { p x + 1 ′ , p x + 2 ′ , ⋯ , p x + y ′ } , x + y = n , x < y .

As the requirement for m kinds of spare parts is predicted based on the consumption data of m kinds of spare parts, the number of neural units in the input layer is m, and the number of neural units in the output layer is also m. The number of hidden layer neural units is s i z e _ h , theoretically the larger the number, the higher the prediction accuracy, but it will increase the amount of network calculations, which can be optimized and determined by the particle swarm algorithm.

3.3 Network training

Set the network training parameters, the number of training cycles MaxEpochs, the number of training data subsets MiniBatchSize, the prediction interval T , and the initial learning rate η . For the training set P t r ′ = { p 1 ′ , p 2 ′ , ⋯ , p x ′ } , take T time data { p 1 ′ , p 2 ′ , ⋯ , p T ′ } initiating from p 1 to calculate the network sequence output { y ˆ ( 1 ) , y ˆ ( 2 ) , ⋯ , y ˆ ( T ) } at time 1 ∼ T through the forward calculation of equations (1)–(7). Then, check { y ˆ ( 1 ) , y ˆ ( 2 ) , ⋯ , y ˆ ( T ) } with { p 2 ′ , p 3 ′ , ⋯ , p T + 1 ′ } at time 2 ∼ T + 1 in the training set. Use the mean square error loss function (13) to calculate the error loss value.

According to the loss function value E t , the reverse transfer algorithm of error data items [22] is used to calculate the adjustment values dU _f , dW _i , dU _i , dW _c , dU _c , dW _o , dU _o , dV, db _f , db _i , db _c , db _o , and db _v of the weights W f , U f , W i , U i , W c , U c , W o , U o , and V and the bias vectors b f , b i , b c , b o , and b v . Continue to take T time data { p 2 ′ , p 3 ′ , ⋯ , p T + 1 ′ } initiating from p 2 to calculate the network timing output value { y ˆ ( 2 ) , y ˆ ( 3 ) , ⋯ , y ˆ ( T + 1 ) } forwardly, then check it with the data { p 3 ′ , p 4 ′ , ⋯ , p T + 2 ′ } at time 3 ∼ T + 2, apply error loss function and reverse transfer algorithm to get the adjustment value of each weight and bias vector, respectively, accumulated with the adjustment value of the previous moment. Follow this method in sequence. After MiniBatchSize times calculation, the weights and bias vectors of the network are updated. The updated value is the cumulative adjustment value multiplied by the learning rate, such as U f = U f + η × d U f , and the adjustment value will return to zero after the update is completed. After MiniBatchSize time data are calculated again, the network weights and bias vectors are updated again. Once the data reaches p x − T + 1 ′ , one training process ends. After repeating MaxEpochs training cycles, the network training is complete. In the training process, the improved Adam optimization algorithm [23] is used to adaptively adjust the learning rate η to make the network training converge as soon as possible, according to the gradient.

3.4 Network prediction

To test the effectiveness of the network, the trained LSTM recurrent neural network and the testing set data P t e ′ = { p x + 1 ′ , p x + 2 ′ , ⋯ , p n ′ } are used to predict data. Starting from p x + 1 ′ , take T time data { p x + 1 ′ , p x + 2 ′ , ⋯ , p x + T ′ } to calculate the output y ˆ ( x + T ) at time x + T according to the equations (1)–(7), continue to take T data from p x + 2 ′ to calculate y ˆ ( x + T + 1 ) until the output y ˆ ( n − 1 ) at time n − 1 is completed. The prediction output set Y = { y ˆ ( x + T ) , y ˆ ( x + T + 1 ) , ⋯ , y ˆ ( n − 1 ) } is de-normalized to obtain the prediction data set P ″ according to equation (13), and the network prediction is completed.

3.5 Parameter optimization

The LSTM recurrent neural network uses the training set data to train the network, optimizes, and adjusts the weight parameters and bias vectors of the input layer, hidden layer, and output layer to obtain a network structure suitable for the prediction object. After the network training is completed, use the testing set data to test the network prediction effect. In the procedure of LSTM recurrent neural network prediction and training, there are four key parameters that affect the quality of the algorithm, which are the number of hidden layer neural units s i z e _ h , the prediction interval T , the number of training data subsets MiniBatchSize, and the initial learning rate η . The influence of these parameters on the algorithm is complicated, and setting by experience is often ineffective. Taking the minimum mean square error of the prediction data set P ″ and the testing set data P t e ′ as the objective function, this article proposes to use the adaptive inertial weight particle swarm algorithm to optimize the four parameters, and use Matlab to implement the algorithm.

4.1 Experimental data and parameter settings

This article takes 48 months consumption data of three main types of maintenance spare parts of a maintenance organization as an example to show the procedure of prediction. The consumption of three types of spare parts is shown in Figure 4.

Figure 4

Consumption of three types of spare parts in a maintenance organization.

The particle swarm algorithm is used to optimize the LSTM recurrent neural network to predict the requirement of equipment maintenance spare parts. The main initial parameter settings of the algorithm are shown in Table 1. Among them, in order to reduce the calculation time, the network training times is set as 100 when using the particle swarm to optimize the LSTM recurrent network parameters, and is set as 300 to enhance the prediction accuracy when using the optimal parameters to construct the network for prediction.

4.2 Simulation result analysis

With the minimum mean square error as the target, the particle swarm algorithm is used to perform 100 iterative calculations, in order to optimize the four key parameters of the LSTM recurrent neural network. The training process is shown in Figure 5. It can be seen that the algorithm converges after 58 iterations, and the optimal mean square error is 3.8524. The four key parameters of the LSTM recurrent neural network are preferably: s i z e _ h = 275, T = 3, MiniBatchSize = 10, and η = 0.3187.

Figure 5

The training process of PSO optimizing the key parameters of LSTM.

The LSTM recurrent neural network is constructed using the four optimal parameters, and the network is trained 1,000 times with 36 months consumption data of three types of spare parts. The values of the loss function in the training process are shown in Figure 6.

Figure 6

Loss function values in LSTM recurrent neural network training.

Figure 7

Legend of Figures 8–11.

It can be seen that the convergence of the algorithm is good, and the loss function value is low. Using 12 months consumption data to test the trained network, the comparison between the predicted value and the actual consumption value is shown in Figure 8, and the mean square error is 3.3611. The error of the spare parts prediction in August is relatively large because the equipment was suddenly used in large quantities during that period, which was inconsistent with the previous two years.

Figure 8

Comparison of LSTM.

Figure 9

Comparison of BP neural network.

Figure 10

Comparison of GRNN.

Figure 11

Comparison of wavelet neural network.

In order to verify the predominant prediction performance of the LSTM recurrent neural network, comparative experiments are performed. Four widely used neural networks currently, BP neural network, generalized regression (GRNN) neural network, wavelet neural network, and squeeze-and-excitation networks (SENet) are used to predict the requirement of spare parts too, using the same training set and testing set. After 1,000 times of network training, 12 months requirement for 3 types of maintenance spare parts was predicted, and the comparison between the 3 kinds of neural network prediction values and actual consumption values were obtained, as shown in Figures 9–12. The mean square errors of the four neural networks are shown in Table 2. From the comparison of the predicted value and the actual consumption value curve and the mean square error value, it can be seen that the LSTM recurrent neural network has the highest prediction accuracy and has obvious advantages in predicting the requirement of maintenance spare parts with time series characteristics.

Figure 12

Comparison of SENet.