Limitations in Evaluating Machine Learning Models for Imbalanced Binary Classification in Spine Surgery: A Systematic Review.

Introduction: ML and DL are being employed to create spine surgery clinical prediction models. These models must be properly evaluated with imbalanced data to be effective in clinical practice. This systematic study identified and evaluated all research-based spine surgery ML and DL models predicting binary outcomes using their assessment measures.

Methods: A comprehensive search of the literature was conducted through the EMBASE, Medline, and PubMed databases using relevant keywords. No limits were placed on the level of evidence or timing of the study. Overall, 60 papers were included and the findings were reported according to the PRISMA guidelines.

Results: Thirteen of the 60 publications focused on LOS, 12 on readmissions, 12 on non-home discharge, six on death, and five on reoperations. Data imbalances ranged from 0.44% to 42.4% for goal outcomes. Fifty-nine articles reported the model's AUROC, 28 accuracies, 33 sensitivity, 29 specificity, 28 PPV, 24 NPV, 25 BS, 10 null model brier, and eight F1 score. The included articles also visualized data differently: 52 had AUROC curves, 27 calibration curves, 13 confusion matrices, 12 decision curves, three PR curves, and one precision-recall curve. Papers claiming good metrics but missing others, maximizing measures at the expense of others, and reporting high accuracy and AUROC but low sensitivity were prevalent errors and bias causes. Papers with poor calibration plots and missing measurements were also present.

Conclusion : Applied machine learning for spine surgery requires proper evaluation techniques. Researchers, reviewers, and editors should understand the dangers of poor evaluation metrics and the relevance of model evaluation. AI models may benefit from expert statisticians' evaluation. This may advance spine surgery research using AI.