Model construction and interpretability analysis of machine learning in risk prediction of cryptogenic stroke in patients with right-to-left shunts

Gao Xiaoli; Tang Sujuan; Tian Feifei; Zhao Hongqin

doi:10.3969/j.issn.1672-5921.2026.02.001

Abstract：Objective To construct an optimal machine learning model for predicting cryptogenic stroke in patients with right-to-left shunt(RLS)and perform interpretability analysis.Methods Retrospectively enroll continuous patients with RLS diagnosed with contrast-enhanced transcranial Doppler ultrasound(c-TCD)foaming experiment for migraine,dizziness(of unknown etiology),stroke,etc.,treated in the Department of Neurology at Laoshan Campus,the Affiliated Hospital of Qingdao University from January 2018 to December 2024.Patient demographics and cardiac parameters from transthoracic echocardiography(TTE)were collected including sex,age,smoking history,alcohol consumption history,hypertension,diabetes,hyperlipidemia and left atrial diameter(LAd),left atrial short diameter(LASd),left atrial longitudinal diameter(LALd),left ventricular end-diastolic inner diameter(LVDd),left ventricular end-systolic inner diameter(LVDs),left ventricular posterior wall thickness(LVPW),interventricular septal thickness(IVS),left ventricular ejection fraction(LVEF),and pulmonary artery systolic pressure(PASP).All patients were categorized into cryptogenic stroke and non-stroke group based on the occurrence of cryptogenic stroke.General characteristics and TTE cardiac parameters were compared between groups.Six machine learning models,including Logistic regression,decision tree,random forest,extreme gradient boosting(XGBoost),gradient boosting,and extreme tree were constructed using general characteristics and cardiac parameters collected from TTE to predict the occurrence of cryptogenic stroke in RLS patients.The performance of the six machine learning models was evaluated using 5-fold cross-validation repeated 10 times.Receiver operating characteristic(ROC)curves were plotted for each model,and the area under the curve(AUC),precision,recall,accuracy,and F1 score were calculated.Calibration curves were plotted for each model to assess whether predicted probabilities approximated true probabilities.Decision curve analysis(DCA)was used to evaluate the net benefit provided by each of the 6 models in predicting cryptogenic stroke in RLS patients within specific threshold probability ranges to determine its clinical value.Threshold ranged 0-＜0.3 was defined as low-risk zone,0.3-0.6 as moderate-risk zone,and>0.6-1.0 representing high-risk zone.The optimal machine learning model was selected based on the best performance across 10rounds of 5-fold cross-validation(primarily using AUC as the metric;when AUC was equal,the model with the highest sensitivity was prioritized).The Delong test was employed to compare the AUC differences between the best-performing model and other models.All patients were randomly divided into training and testing sets at an 8∶2 ratio.The optimal machine learning model was retrained on the training set and then applied to the testing set data for Shapley additive explanations(SHAP)analysis.Core predictors for cryptogenic stroke in RLS patients were identified by descending average SHAP values in feature importance histograms.SHAP scatter plots was plotted to reveal the correlations between feature-value and model outputs.SHAP Force plots was used to test individual patient prediction mechanisms,and to represent each feature's directional and intensity contribution to case-specific decisions through area-based visualization.Partial dependency plots(PDPs)were generated for the top two SHAP-ranked features to evaluate their marginal effects and nonlinear trends in predicting cryptogenic stroke in RLS patients,and explore how the SHAP tree model quantified the relationship between key predictors and risk.Results A total of 310 patients with RLS were included,comprising 181 males and 129 females aged 18 to 70 years with median of 52(43,52)years.Among them,164 patients was categorized into the cryptogenic stroke group and 146 to the non-stroke group.(1)Compared with the non-stroke group,patients with RLS in the cryptogenic stroke group had significantly higher age,more male patients,and higher proportion of patients with hypertension,diabetes hyperlipidemia,smoking history,and alcohol consumption history.Additionally,they demonstrated significantly larger LAd,LASd,LALd,LVDd,LVDs,LVPW,IVS,and PASP(all P＜0.05).No statistically significant differences were observed in LVEF between groups(P=0.306).(2)ROC curve analysis of six machine learning models predicting cryptogenic stroke using RLS,evaluated via 5-fold cross-validation with 10 repetitions.From the analysis,the extreme tree model had the highest average AUC(0.804)in the prediction of cryptogenic stroke in RLS patients,with random forest(0.792)ranked second,and in descending order,gradient boosting(0.777),XGBoost(0.776),Logistic regression(0.764),and decision tree(0.717).(3)Calibration curve results indicated that the extreme tree model's calibration curve(Brier score0.197)most closely approximated the reference line,followed by random forest(Brier score 0.210),Logistic regression(Brier score 0.222),gradient boosting(Brier score 0.244),decision tree(Brier score 0.263),and with the XGBoost(Brier score 0.280)performing worst.DCA revealed that within certain medium probability threshold ranges(0.3-0.6),the extreme tree model's DCA curve demonstrated higher net gain rates,showing significantly better performance than other models at specific points.At high probability thresholds(>0.6-1.0),the net gain rate decreased.At low probability thresholds(0-＜0.3),the net gain rate was low and approached the non-intervention curve,with the difference narrowing.Among the six machine learning models,the extreme tree model performed best,achieving an AUC of 0.804,accuracy of 0.719,and F1 score of 0.736.Delong's test revealed a statistically significant difference in AUC between the extreme tree and decision tree models(P＜0.05),while no significant differences were observed with other machine learning models(all P>0.05).(4)Among 310 RLS patients,248 were in the training set and 62 in the test set.Compared with the test set,RLS patients in the training set had significantly higher PASP(P=0.037).No statistically significant differences were observed between the training and test sets in general characteristics or TTE cardiac parameters(all P>0.05).(5)The extreme tree model was retrained based on the training set data and tested with data in the testing set.An SHAP analysis was then performed on this result.The SHAP analysis showed that LAd was the most important feature,followed by LALd,age,LVPW,LVDs,hypertension,LVDd,smoking history,and PASP.The SHAP scatter plot indicated that the top two factors potentially increasing the risk of cryptogenic stroke in RLS patients were LAd and LALd.(6)The Force plot of the decision tree model showed that when the patient's SHAP value was 1.0,the age was 68years old,and the corresponding LAd,LALd,LVDd,IVS,and LVDs were 34,56,48,10,and 30 mm,respectively.Among these,the LALd region had the largest proportion,while the LVEF region had the smallest proportion.(7)The partial dependency plot generated based on the top two feature importance values of LAd and LALd from the extreme tree model showed that,when LAd＜32 mm,the prediction probability was around 0.40;when LAd was between32-42mm,it increased from0.35 to0.60,if LAd>42mm,the growth rate of slows,with an average prediction probability value of 0.65.While,if LALd＜40 mm,the prediction probability remains stable below 0.40;if LALd was between 40-55 mm,it steadily increased to nearly 0.60;and the growth rate slowed when LALd>55 mm.Conclusions The extreme tree model constructed using general and cardiac parameters of RLS patients can make relatively accurate prediction of the risk of cryptogenic stroke in patients with RLS,with LAd and LALd ranked as the top two factors influencing the model's output.The model's validity in external cohorts and its potential for translating predictive value for cryptogenic stroke risk remain to be determined.