Evaluating The Generative Pretrained Transformer Model’s Proficiency in Neuroradiology: A Binomial Analysis

Introduction: As the medical field explores the potential of large-scale transformer models in imaging, neuroradiology question banks emerge as a distinct avenue for evaluating models like the Generative Pretrained Transformer-4 (GPT-4). Unlike traditional imaging datasets, these question banks provide both images and contextual information, simulating the integral role of neuroradiology in neurosurgical decision-making. In light of this unique context, our study sought to evaluate the proficiency of GPT-4 on neuroradiology question banks.

Methods: Neuroradiology question banks were sourced from Radiology Masterclass (n=20) and LearnNeuroRadiology (n=57) in sets of 9 and 10 and assessed by GPT-4. To gauge the model's performance against chance, we employed a binomial test, benchmarking against the probabilities expected from random guessing: 0.20 for Radiology Masterclass and 0.25 for LearnNeuroRadiology.

Results: For Radiology Masterclass’ quiz 1, GPT-4 achieved an accuracy of 40% while for quiz 2, GPT-4 achieved an accuracy of 60%. Aggregating results from both quizzes, GPT-4 achieved an overall accuracy of 50% on this question bank, significantly outperforming random guessing with a p-value of 0.0026 and statistical power of 99.66%. In a subsequent evaluation of the LearnNeuroRadiology dataset, GPT-4 achieved accuracies of 55.56%, 60%, 40%, 44.44%, 60%, and 33.33% for the quizzes in sequence. Aggregated results demonstrated a 49.12% accuracy, with a binomial test p-value of 7.34×10−5 and 100% statistical power.

Conclusion : By leveraging the unique combination of images and contextual clues provided by neuroradiology question banks, GPT-4 demonstrated a proficiency that significantly surpassed random guessing across two distinct datasets. As the field of neurosurgery continues to evolve, the synergy between AI and neuroradiology may pave the way for enhanced diagnostic accuracy, timely interventions, and improved patient outcomes.