Hierarchical reinforcement learning-based optimization method for crane scheduling
SU Zhang-sheng
JIANG Sheng-long
PENG Gong-zhuang
LIANG Yue-yong
Abstract：Cranes are key heavy-duty material handling equipment widely used in shops,warehouses,ports,and other industrial settings.The scheduling of cranes significantly affects transportation efficiency and the achievement of produc-tion goals.To address the crane scheduling problem with time windows(CSP-TW),a mixed-integer linear programming model based on spatio-temporal discretization is developed.Based on the characteristics of the model,a hierarchical rein-forcement learning(HRL)decision-making framework is designed.The high-level decision network assigns transportation tasks to appropriate cranes,while the low-level network plans paths for each crane to complete its assigned task.During the learning process,action tabu rules are introduced to avoid ineffective actions and guide the decision networks toward the dominant policy space.Subsequently,external experience pooling and the dueling double deep Q-network strategy are adopted to train the decision networks.Tests were executed based on the logistics simulation platform of a steel plant from a certain company.Ablation experiments show that the introduction of action tabu rules improves learning efficiency.Training comparisons indicate that HRL achieves better convergence than the end-to-end framework.Compar-ative experiments demonstrate that HRL outperforms several methods,including multi-rule combinations,meta-heuristic algorithms,end-to-end and deep Q-network,while satisfying second-level response-time requirements for applications.
Keywords：crane schedulinghierarchical reinforcement learningtask assignmentpath planningaction tabu
Publication Date：2025-11-30
Online Publishing Date：2025-12-29(First online date of this platform, not the publication date of the document)
Pages：13( 2261-2273 )