Can Artificial Intelligence Pass the Neurosurgery Board Exam? Assessing Popular Artificial Intelligence's performance on the American Board of Neurological Surgery Review Questions

Introduction: Previous studies have demonstrated that ChatGPT has the ability to successfully answer neurosurgery boards review questions, however ChatGPT’s ability to analyze images and other publicly available AI platforms were not appraised. In this study the authors sought to assess the ability of popular AI platforms to answer American Board Neurological Surgery (ABNS) questions, as well as investigate ChatGPT’s ability to analyze image based questions.

Methods: Questions were sourced from “The Comprehensive Neurosurgery Board Preparation Book” by Paul Byrini, MD. Popular AI platforms were asked the questions once at a time in a multiple choice format. Questions that required images were excluded from all platforms except ChatGPT-4. Statistical analysis was performed with Python (Python Software Foundation, Wilmington, DE).

Results: The practice test consisted of 375 questions. For questions without images (282), the accuracies were: ChatGPT-4 75%, BingAI 66%, ChatGPT3.5 62%, and Perplexity.AI61%. ChatGPT-4 outperformed ChatGPT3.5 and Perplexity.AI but not BingAI (F=20.92, p< 0.001). For questions with images (93), ChatGPT-4 answered 45% correctly, leading to an overall score of 67%. ChatGPT-4’s performance on nonimage based questions was superior to questions with images (75% v. 45%, p< 0.0001). There was no observed difference in performance of the category of questions if they required an image or not (F(9, 272) = 0.881, p = 0.542).

Conclusion : Considering the historical percentage correct of the ABNS Primary Exam that corresponded to a passing score prior to 2022 was approximately 62%, our findings suggest that popular AI may mirror exam performance to that of a neurosurgery resident studying for the ABNS Primary Exam. ChatGPT-4’s performance was superior to other platforms apart from BingAI. However, it is important to note that BingAI, Perplexity.AI, ChatGPT3.5 do not require subscriptions and can be a free resource for those studying for the exam. While ChatGPT-4’s performance on image based questions was poor, it did not preclude the platform from achieving a likely passing score.