The Digital Advantage: Using a Large Language Model and Machine Learning to Evaluate Neurosurgery Residency Applicants

Introduction: The selection of neurosurgery residents is paramount yet arduous, with the average application containing more than 58 pages of information. We aimed to create a machine learning model to predict the probability of applicants getting interview invites and matching into neurosurgery based on pre-interview application features.

Methods: We retrospectively analyzed applications to an ACGME-accredited neurosurgery residency program for the 2022-2023 application cycle. Using a large language model, Python scripts were developed for a pipeline processing applicant PDFs, extracting data such as demographics, education, AOA status, publication count, and program evaluations, alongside ERAS-provided USMLE scores and basic demographics.

Results: From 279 applications and 16,273 pages, we found that higher USMLE Step 1 scores and AOA status were significant predictors of an interview invitation (p = 0.0057, p = 0.0001). US seniors were likelier to get interviews over other US graduates, International Medical Graduates (IMG), or Doctor of Osteopathic Medicine (DO) degree holders (p = 0.001), and were significantly more likely to match (p = 2.35 × 10^-13). Higher scores in home and away program evaluations also correlated with match success (p = 0.062, p = 0.001). A neural network was 89% accurate in predicting interviews, while ridge regression and random forest models had 90% accuracy for match likelihood. However, including interview scores in models like XGBoost decreased predictive accuracy. Key positive features were program evaluations, US senior status, and USMLE scores, while lack of evaluations, non-US senior status, absence of USMLE Step 2 CK score, and being Black or African American reduced chances.

Conclusion : We successfully utilized a language model to build a machine-learning pipeline for the evaluation and prediction of neurosurgery applicants' interview and match likelihoods. While the model may be a useful adjunct in evaluating candidates, the results highlight the need to consider implicit racial biases when evaluating neurosurgery residency candidates.