A study on deep reinforcement learning-based crane scheduling model for uncertainty tasks

3.1 Basic principles of deep reinforcement learning

Deep reinforcement learning is a method that combines the perceptual ability from deep learning and decision-making from reinforcement learning. It can realize a systematic and integrated intelligent algorithm framework from perception to decision-making.

Deep learning adopts convolutional neural network, it is a type of feedforward neural network which contains convolutional calculation and deep structure, normally composed of convolutional layer, pooling layer, and full link layer. In reinforcement learning, intelligent agents with learning ability have interactive trial and error with their environment, allowing the continuous learning and achievement of the objective. General process of reinforcement learning is as below:

1. The intelligent agent observes the status of environment s _t at moment t and selects action a _t based on current strategy π;

2. Affected by action a _t , the environment status changes to s _t+1, assessment on the selected action of the intelligent agent is carried out and reward r _t is given based on one type of reward function R;

3. The intelligent agent adjusts the strategy according to the reward feedback, and the process returns to step (1).

The intelligent agent finally obtains an optimal strategy π* by constantly repeating these steps, and reaches the highest accumulated reward value by end of the cycle.

The Q-learning algorithm adopted in deep learning is the classic algorithm in computing optimal strategy π*. The core is the status-action value function Q ^π (s, a) through learning and iterating. The function describes the value of each action a under status s. When achieving the optimal function Q ^π (s, a), the optimal strategy refers to the action at that leads to maximum value of Q π ⁎ ( s t , a t ) under status s _t . The iteration and learning of Q π ⁎ ( s t , a t ) is shown as below:

where k is the times of iteration, s _t is the status at moment t, a _t is the action at moment t, r _t is the immediate reward executing action a _t under status s _t . Similarly, s t + 1 and a t + 1 refer to the status and action at the next moment. α ∈ ( 0 , 1 ) is the learning rate and γ ∈ ( 0 , 1 ) is the reward discount factor.

3.2 Deep reinforcement learning-based crane scheduling model

The deep reinforcement learning-based crane scheduling model views each crane as single intelligent agent and the respective workshop as the environment. The action of the intelligent agent is the action of crane, the status it observes is the status of task information and status of all cranes within the cross-region. Instructed by reward function that is designed through objective function, cranes can complete the task highly efficiently. The formation process of reinforcement learning mechanism is shown as Figure 1.

1. Environment is the scene that the intelligent agent interacts within, it is normally an abstraction of the actual environment, including the status space, action space, and reward function.

Figure 1

Reinforcement learning mechanism of crane scheduling model.

Status space is the set of potential status the environment can occur, which needs to contain all feature information necessary for decision-making. To reduce the complexity of status space, for crane scheduling problem, the cross-region space is abstracted to a two-dimensional matrix. The status space is symbolized through the status information marked by different values in the matrix. This study selected a 3 × 30 matrix to symbolize status space. The first row indicates the beginning position of transportation task, the second row refers to the numbering and status information of cranes, and the third row shows the ending position of the transportation task. Sample of the status space matrix is shown as Figure 2.

Figure 2

3 × 30 matrix of status space in crane scheduling.

Such design of status space neglects the crane movement along the main beam direction, since such movement takes much less time and distance compared to the orbital direction and therefore, barely has any impact on crane scheduling.

Action space is the set of selectable actions generated by the interaction between intelligent agents and environment. In crane scheduling, cranes normally have seven actions, which are moving left/right along orbital direction, moving forward/backward along main beam direction, standstill, lifting, and putting down. Since the selected status space in this study has neglected the movement along main beam direction, the action space is simplified to 5 instead of 7, represented by 0, 1, 2, 3, and 4, respectively.

Reward function is the assessment rule for the actions of intelligent agent. It is an important instruction signal for the intelligent agent to learn and improve and should be designed according to the objective function of the problems. To obtain a practical and efficient crane scheduling model, minimum task completion time is set as the objective and the reward function is designed as formula (2).

where f ( t ) is the function negatively correlated to the task completion time t , which means, when t is smaller, and f(t) is higher.

As the reward function is designed to have reward only after completing the task, there can be a problem of reward sparsity. Therefore, intensive improvement for reward function is required, that is to supplement some reward value to the five actions left/right movement along the orbital direction, standstill, lifting, and putting down. In the meantime, add negative reward for abnormal situations like collision and moving out of the reasonable range.

1. Intelligent agent consists of neural network structure and reinforcement learning algorithm.

Neural network structure adopts convolutional neural network in deep learning. For crane scheduling issue, the 3 × 30 matrix of status space in crane scheduling is the input for convolutional neural network. The five actions in action space are its output. Detailed neural structure is shown in Figure 3.

Figure 3

Sketch of neural network structure of intelligent agent.

The convolutional neural network includes two convolution layers and two full link layers. The convolution kernel has a size of 3 × 3, the number of neurons in full link layer is 1,024, and activation function uses the ReLU function.

In crane scheduling, specific convolutional neural network needs to be designed for each crane. How to coordinate multiple neural networks to complete the training simultaneously is the key in the research of reinforcement learning algorithm in crane scheduling. This study made improvement in redesigning the interaction between multiple Q network and the environment on the basis of Deep Q-Learning (DQN) algorithm. Taking two cranes as example, the interaction is shown as Figure 4.

Figure 4

Interaction mode between Q network and environment (2 cranes).

In Figure 4, 1–9 is the Q network training step sequence for two cranes; s t 1 is the environment status of crane 1 observed in moment t; s t 2 is the environment status of crane 2 observed in moment t; s t + 1 1 is the environment status of crane 1 observed in moment t + 1; a t 1 is the action of crane 1 in moment t; a t 2 is the action of crane 2 in moment t; r t 1 is the immediate reward to crane 1 in moment t; r t 2 is the immediate reward to crane 2 in moment t; r _t is the final immediate reward to two cranes in moment t.

The training process of multi-crane is similar to that of two cranes. Within one unit of time, multiple cranes select one action according to the environment and act upon it sequentially. The sum of respective reward value is the immediate reward of multiple network models. When there are conflicts in the routes, they will choose the optimal avoiding method to get the highest reward value, since the immediate reward is the sum of rewards these cranes can get individually.

3.3 Parameter optimization in crane scheduling model

To increase the training efficiency of intelligent agent under the framework of deep reinforcement learning, optimization is required in reward discount factor γ, learning rate α , and reward function according to key indexes of convergence rate and training time in the uncertainty task scheduling problem in multi-crane situation.

In order to optimize the parameters in one unified environment, an integrated basic situation of multi-crane scheduling is constructed. Taking five successive transportation tasks as the basic unit of crane scheduling, a transportation task set consisting of 46 kinds of different scheduling units is created.

1. The reward discount factor γ is a factor influencing intelligent agents’ attention degree on future reward. The value range is 0–1 and the normal span is 0.5–0.9. This study tests and compares the convergence rates of deep reinforcement learning based crane scheduling model at the γ value of 0.5, 0.6, 0.7, 0.8, and 0.9 for models with different parameters reaching the training goal, the convergence rate is represented by the ratio of completing transportation tasks under the same training step. The comparison result is shown in Figure 5.

   It can be seen from Figure 5 that when γ equals 0.8, the training speed is the fastest. Hence, 0.8 is chosen as the value of γ for multi-crane scheduling model.

2. Learning rate α determines the magnitude of each parameter updated in Q network model. When the value is relatively low, the model training is stable but slow. When the value is too high, the result can be volatile and cannot converge. Considering the possibility of random coupling between environment status and action, a relatively stable training process is necessary to ensure that the model inclines to converge. Therefore, the value of learning rate α is optimized within the range of 0.0001∼0.001. This study tests and compares the convergence rate when α equals to 0.0001, 0.0003, 0.0005, 0.0007, and 0.0009. The result is shown in Figure 6.

   It can be seen from Figure 6 that the training speed is fastest when α equals 0.0005. Hence, 0.0005 is adopted as learning rate in multi-crane scheduling model.

3. The basic form of reward function is designed as formula (2) in Section 3.2. Intensive improvement is necessary to solve the reward sparsity issue in crane scheduling. Original, weak, and strong reward mechanisms are designed, tested, and analyzed through scheduling process analysis and experts’ consultation in different areas. The design of three models are displayed in Tables 1–3.

Figure 5

Convergence rate comparison for models with different reward discount factors.

Figure 6

Convergence rate under different learning rates.

Based on the original reward mechanism, on the one hand, strong reward mechanism and weak reward mechanism further subdivide the crane status and set different reward values for different crane actions. On the other hand, according to the principle of multiple scaling, weak reward mechanism narrows and strong reward mechanism enlarges the action reward value that involves transportation tasks. Through the setting of three reward mechanisms, the influence of different reward mechanisms on the model training process is tested.

With the goal of randomly extracting four groups of basic unit task in crane scheduling, the training time and total task completion time are collected and results displayed in Table 4 under the above three reward mechanisms.

From Table 4 it can be concluded that the training time reduced significantly after intensive improvement in reward mechanism. Considering the total task completion time comprehensively, strong reward mechanism is selected as the training parameter for crane scheduling model.

4.1 Experiment data

Crane scheduling situations involving both multi-crane scheduling and significant uncertainty mainly occur in industrial production processes. Taking steelmaking process as an example, since there are multiple smelting production lines within one steelmaking workshop, and multiple cranes are shared to complete transportation tasks between the production procedures, it is a typical multi-crane scheduling situation. In the meantime, as the smelting time of production procedure is volatile to some extent, the crane tasks have apparent uncertainty, further increasing the difficulty of crane scheduling.

The crane scheduling process of steel span in an actual steelmaking workshop is selected as the situation to test and verify the proposed model. The layout of the span is shown in Figure 7. The total length of the span is 300 m, equipped with 2 converters, 2 double-position RH refining furnaces, 1 double-position LF refining furnaces, 1 double-position CAS refining furnaces, 2 continuous casting machine, 2 baking stations, 2 thermal repair stations, 1 dumping position, and 2 cranes.

Figure 7

Layout of steel span in one steelwork.

The transportation tasks in the span include two phases: heavy ladle (loaded with molten steel) transport and empty ladle (without molten steel) transport. Heavy ladle transport consists of molten steel transported from converter to refinery, from refinery to continuous casting, and some from LF refining to RH refining. Empty ladle transport includes the slag turning activity after continuous casting, transported from dumping position to thermal repair position, from thermal repair to baking station, and from baking station to converter position.

In total, 10,900 items of crane tasks are collected and filtered according to time sequence, based on the actual production data and crane operation data in the workshop between January and March 2016. Some crane task data are shown in Table 5. Taking the first 10,000 items of data, set every 5 consecutive tasks as one basic unit, and form the basic unit set of training model. Take the latter 900 items of data and randomly intercept every 100 consecutive tasks as the testing set of the model. Then, test and verify the executive effect of the crane scheduling plan generated by the model.

4.2 Deep reinforcement learning based crane scheduling plan

This study constructed the deep reinforcement learning-based crane scheduling model for uncertainty tasks under multi-crane situation by applying Python 2.7 language, deep learning framework Pytorch 0.2.0, and reinforcement learning platform OpenAI Gym. The major parameters of the model are learning rate α = 0.0005, reward discount factor γ = 0.8, and playback memory pool size M = 1 million, and a strong reward mechanism is adopted as the intensive strategy of reward function.

Based on historic task data, the ratio that the model completes task sequence gradually becomes stable and close to 100% convergence rate after 9 million steps of iterative training. The convergence curve of the training process is displayed in Figure 8.

Figure 8

Convergence curve of training process in crane scheduling model.

The crane scheduling plan for testing set is created based on the model and the time-space orbit graphics of the cranes are drawn, some graphics are shown in Figure 9.

Figure 9

Some time–space orbit graphics of cranes.

From the analysis of the graphics, it can be seen that there is no collision or moving out of the orbit range during the crane operation process. In the meanwhile, when comparing the task completion time in the scheduling plan and the target completion time in the task list, all tasks are completed within the target time. Based on these results, it can be concluded that the scheduling plan can effectively complete the defined crane tasks and the feasibility of deep reinforcement learning-based crane scheduling method is verified.

4.3 Comparison between deep reinforcement learning-based scheduling plan and manual scheduling plan

Currently there is no crane scheduling model that can be applied in practical production process, due to the uncertainty in crane tasks and the difficulty in dynamic coordination between multiple cranes in steelmaking workshops. In all steelmaking workshops, manual command is applied, allocating cranes to transportation tasks dynamically and coordinating the routes planning between multiple cranes.

To examine the high effectiveness of the deep reinforcement learning-based crane scheduling model proposed by this study, the crane scheduling plan generated by the model is compared with the manual scheduling plan in actual production process (with same 100 items of crane tasks). The advantages and disadvantages of two scheduling plans are analyzed and assessed from three dimensions: task completion time, times of routes collision, and return distance when routes collide. The comparison result is shown in Table 6.

It can be observed that, compared to manual scheduling plan, deep reinforcement learning-based crane scheduling plan shortened the task completion time by 1,405 s or 11.52%, which indicates that the transportation task can be completed within a shorter time. The times of routes collision decreased by 12 or 57.14%, passive crane transportation distance reduced by 420 m or 55.26%, proving that the proposed method has more reasonable route planning and lower risk of crane collision, thereby increasing the efficiency of cranes.

All the above comparisons indicate that, through reasonable design of deep learning network and reinforcement learning mechanism, and optimization of parameter setting, especially intensive improvement in reward function, the deep reinforcement learning-based crane scheduling model can quickly grasp the scheduling strategy of completing task and reducing route collision, thereby generating a more effective and efficient crane scheduling plan.

Compared with manual scheduling method, crane scheduling method based on deep reinforcement learning can explore potential data patterns through iterative training of transportation task data-set, and form a more reasonable and efficient scheduling plan than human experience. The main reason is that, on the one hand, the transportation task data-set combined with actual production data contains more comprehensive task types and composition relationships, and its amount of information far exceeds the memory and experience of dispatchers. On the other hand, the method based on deep reinforcement learning can approach the global optimum scheduling plan through continuous iterative training. The space span, time span, and task volume considered by the scheduling plan far exceed the capability of the dispatchers.