Study on Tongue Coating Classification Method Based on Joint"Image-Spectrum"Features

ZHANG Yahan; CHUN Yi; CUI Longtao; TU Liping; JIANG Tao; HU Xiaojuan; CUI Ji; QIU Xipeng; XU Jiatuo

Abstract：Objective This study aims to integrate tongue coating image features with spectral information,utilizing machine learning models to enhance the accuracy and reliability of tongue coating classification,thereby providing digital support for traditional Chinese medicine(TCM)tongue diagnosis.Methods The"TFDA-1 Tongue Diagnosis Device"was used to capture tongue image features,and the PG2000-PRO-EX fiber optic spectrometer collected spectral data.After data processing,1023 tongue samples were obtained:692 white coatings,261 yellow coatings,37 gray-black coatings;358 greasy coatings,473 thick coatings,455 thin coatings,and 95 peeled coatings.A deep learning model combining ResNet50V2(for images)and Conv1D(for spectra)was constructed for feature fusion and classification.Gradient-weighted Class Activation Mapping(Grad-CAM)visualized key image regions,while random forest evaluated spectral band importance to reveal the role of image and spectral information in tongue coating classification.Results The image-only model achieved accuracies of 0.869(greasy coating)and 0.807(thickness classification).With spectral fusion,accuracy improved to 0.935 and 0.895,respectively,indicating that feature layer fusion enhanced the classification performance of these two types of features.The classification performance of tongue coating color did not significantly improve after introducing spectral information,with an accuracy of 0.864.Grad-CAM shows that the key areas for tongue coating color,greasy coating discrimination,and thickness classification are concentrated on the coating surface,tongue edge,and outer side or peeling of the tongue.The importance score of random forest shows significant differences between white coating,yellow coating,and gray black coating at 380 nm-510 nm.The overall reflectivity of greasy coating is relatively high,and gradually increases from peeling to thick coating at 400 nm-600 nm.Conclusion The multimodal integration of image and spectral data with deep learning significantly improves the classification of greasy coatings and thickness,advancing objective TCM tongue diagnosis.