Balancing Readability and Content Fidelity: Assessing GPT-4.0's Simplification of Neurological Education Materials

Saturday, May 4, 2024

4:08 PM – 4:13 PM CT

Location: S402ab

Presenting Author(s)

Bryce Picton, B.S.

Medical Student
UC Irvine School of Medicine
Orange, CA, US

Co-Author(s)

SA

Saman Andalib, B.S.

Medical Student
UC Irvine School of Medicine, United States
BC

Brandon Camp, B.S.

Medical Student
UC Irvine School of Medicine, United States
JC

Jefferson Chen, n/a

Neurosurgeon
University of California Irvine
Orange, California, United States
PC

Patrick Chen, MD

Assistant Clinical Professer of Neurology
University of California, Irvine, United States
FH

Frank P.K. Hsu, MD, PhD

Chair of the UCI Department of Neurosurgery
Department of Neurological Surgery, University of California, Irvine, UCI Medical Center, Orange, CA
Orange, California, United States
JL

Jason Liang, B.S.

Medical Student
University of California, Irvine School of Medicine, United States
MO

Michael Oh, MD

Clinical Professor of Neurological Surgery
University of California, Irvine
Orange, California, United States
SS

Sean Solomon, BSc

Medical Student
UC Irvine School of Medicine, United States
AS

Aidin Spina, B.S.

Medical Student
UC Irvine School of Medicine, United States

Disclosure(s):

Bryce Picton, B.S.: No financial relationships to disclose

Introduction: The escalating complexity of medical literature necessitates tools to enhance readability for patients. This study aimed to evaluate the efficacy of ChatGPT-4 in simplifying neurology and neurosurgical abstracts and patient education materials (PEMs) while assessing content preservation using Latent Semantic Analysis (LSA).

Methods: A total of 100 abstracts (25 each from Neurosurgery, Journal of Neurosurgery, Lancet Neurology, and JAMA Neurology) and 340 PEMs (66 from the American Association of Neurological Surgeons and 274 from the American Academy of Neurology) were transformed by a GPT-4.0 prompt requesting a 5th grade reading level. Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FKRE) scores were used before/after transformation. Content fidelity was validated via LSA (ranging 0-1 with 1 meaning identical topics) for all texts and by expert assessment (0-1) for a subset (n=40). Pearson correlation coefficient compared assessments.

Results: FKGL decreased from 12th to 5th grade for abstracts and 13th to 5th for PEMs (p <.001). FKRE scores showed similar improvement (p <.001). LSA confirmed high content similarity for abstracts (mean cosine similarity 0.746) and PEMs (mean 0.953). Expert assessment indicated a mean topic similarity of 0.775 for abstracts and 0.715 for PEMs. The Pearson coefficient was 0.598 for abstracts and -0.167 for PEMs. Segmented analysis revealed correlation >.50 below 450 words and a negative correlation above 450 words.

Conclusion : GPT-4.0 markedly improved the readability of medical texts, predominantly maintaining content integrity as substantiated by LSA and expert evaluations. LSA emerged as a reliable tool for assessing content fidelity within moderate-length texts, but its utility diminished for longer documents, overestimating similarity. These findings support the potential of AI in combating low health literacy, however, the similarity scores indicate expert validation is crucial to ensure critical details have not been lost. Future research must strive to improve transformation precision and develop validation methodologies that are both rapid and accurate.