Evaluation of large language models in the field of occupational health

WANG Qi-ting; CAO Bo-yu; LIN Hao-yun; ZHANG Hao-bin; CHENG Wei-bin; HUANG Yong-shun

doi:10.13820/j.cnki.gdyx.20253314

Abstract：Objective To evaluate the knowledge-based question answering and case reasoning capabilities of mainstream Chinese large language models(LLMs)in occupational health and to explore their potential as tools for work-ers' occupational health information access.Methods Seven LLMs,including DeepSeek-R1,DeepSeek-v3.2,Dou-bao-Seed-1.6,Doubao-Seed-1.6-Thinking,GPT-4o,Kimi-k2,and Qwen3-Max,were assessed using a test set of 842 objective questions covering laws and regulations,occupational health monitoring,basic occupational health knowledge,occupational disease diagnosis,and integrated exercises,as well as a case set of 20 real-world occupational health scenarios containing 46 questions.Each model was tested independently for three rounds,and mean accuracy with standard deviation was calculated.Pearson x2 test compared performance across question types,paired t-test evaluated chain-of-thought reasoning efficacy,and case answers were scored by professionals.Results Except for GPT-4o(48.58％±7.12％),all models achieved＞70％mean accuracy,with DeepSeek-R1 performing best(75.79％±5.82％).Significant differences in accuracy among question types were observed(P＜0.05),following the trend single-choice＞true/false＞multiple-choice.Chain-of-thought models outperformed non-chain-of-thought models(t=10.69,P＜0.001).In case reasoning,Kimi-k2(80.28％±15.81％)and Doubao-Seed-1.6(78.82％±14.54％)showed superior performance,and models generally achieved higher accuracy on occupational disease diagnosis than on occupational law questions.Conclusion Mainstream Chinese LLMs demonstrate moderate accuracy in occupa-tional health question answering and real-world case reasoning,offering a technically feasible approach to improve work-ers' access to occupational health information.LLMs hold promise as intelligent tools to enhance occupational health litera-cy and reduce informational barriers.