Can Artificial Intelligence Mitigate Missed Diagnoses by Generating Differential Diagnoses for Neurosurgeons?

Introduction: Neurosurgery emphasizes the criticality of accurate differential diagnoses, with diagnostic delays posing health and economic challenges. As artificial intelligence (AI) emerges as a transformative tool in healthcare, this study seeks to elucidate its role in assisting neurosurgeons with the differential diagnosis process.

Methods: This study employed three chat-based AI tools, ChatGPT (versions 3.5 and 4.0), Perplexity AI, and Bard AI, to evaluate their diagnostic accuracy. Each AI was prompted using clinical vignettes, and their responses were recorded to generate differential diagnoses for 20 neurosurgical disorders, which were identified using ChatGPT. Disease-specific prompts were crafted using Dynamed, a clinical reference tool. The accuracy of the LLMs was determined based on their ability to identify the target disease within their top differential diagnoses correctly.

Results: In evaluating LLMs for differential diagnosis generation, ChatGPT versions 3.5 and 4.0 consistently outperformed other models across various metrics. For the initial differential, ChatGPT 3.5 achieved an accuracy of 52.63%, closely followed by ChatGPT 4.0 at 53.68%, while Perplexity AI and Bard AI lagged at 40.00% and 29.47%, respectively. As the number of considered differentials increased from two to five, ChatGPT models maintained their lead, with ChatGPT 3.5 reaching its peak accuracy of 77.89% for the top five differentials. In contrast, Bard AI and Perplexity AI showed varied performances, with Bard AI improving in the top five differentials at 62.11%. On a disease-specific note, the LLMs excelled in diagnosing conditions like epilepsy and cervical spine stenosis but faced challenges with more complex diseases such as Moyamoya disease and ALS.

Conclusion : AI, especially the ChatGPT models, showcases the potential to enhance diagnostic accuracy in neurosurgery. While promising, the practical implementation of AI in neurosurgery requires a thorough evaluation, considering its limitations and the critical nature of the field.