Multimodal fusion method for foreign object open-set detection in raw coal sorting scenario
CAO Xiangang
LIU Hang
LIU Jiahui
WU Xudong
WANG Peng
Abstract：In the raw coal sorting process,large gangue blocks,steel wires,woven bags,and other foreign objects are first identified and removed to avoid adverse impacts on subsequent processing stages or the occurrence of safety accidents.Current coal foreign-object detec-tion algorithms are primarily designed for closed-set scenarios and focus on detecting known object categories,exhibiting limited capabil-ity in detecting and recognizing unknown-category targets-particularly anchor rods,novel support materials,and other objects with com-plex appearances and semantic uncertainty.Consequently,there is an urgent need to investigate object detection models that can simultan-eously handle both known and unknown foreign objects.Firstly,a text-image bimodal feature extraction architecture is constructed based on the DINO network,and the Path Aggregation Feature Pyramid Network(PAFPN)is introduced.A multi-layer feature extraction strategy is adopted to enhance the perception of small-scale foreign objects.Secondly,a multimodal feature fusion module based on the self-attention mechanism and cross-attention mechanism is built,and a language-guided query selection mechanism is incorporated to achieve deep interaction between text and visual features,thereby improving the semantic consistency of features and cross-category gen-eralization ability.Finally,a vision-text multimodal decoding module is designed,which inserts a text guidance mechanism at the query update stage of each layer to improve the accuracy and robustness of multimodal feature alignment.An open and dynamic environment with multi-category combinations was constructed based on a self-built coal foreign object dataset,and systematic experiments are conduc-ted.The results demonstrate that the proposed method outperforms all baselines in mAP@0.5 on known-category detection across open-ness levels,and attains unknown-category recall rates of 41.24％,52.26％,and 57.13％on unknown-category detection,confirming its zero-shot effectiveness.The method proposed in this paper demonstrates effectiveness in detecting foreign objects in unknown types of coal,providing effective technical support for the open-set detection of foreign objects in coal.
Keywords：coal foreign objectsmultimodal fusionopen-set detectionfeature pyramidsemantic consistency
Publication Date：2026-01-31
Online Publishing Date：2026-03-12(First online date of this platform, not the publication date of the document)
Pages：11( 464-474 )