Using a task based transfer learning prompt generative approach to unsupervised segmentation for AI-enabled neurosurgical guidance

Introduction: A major challenge in leveraging Artificial intelligence (AI) and computer vision (CV) for Surgeon-Machine Interfaces (SMIs) that provide real-time anatomical guidance and decision support is the need for extensive training and annotation data to ensure accuracy and reliability. While the Segment Anything Model (SAM; Meta Inc.) offers a promptable segmentation model for zero-shot generalization, its application in literature has been limited due to suboptimal segmentation. We present a proof-of-concept CV architecture, utilizing task-based transfer learning and unsupervised segmentation for surgical scene analysis. This approach shows promise as a procedure-agnostic segmentation backbone for widespread adoption of SMIs in neurosurgery, even in resource-constrained settings.

Methods: Our architecture utilized SAM with zero-shot prompting through the AdaptIS network. We performed retrospective experiments on spinal dural arteriovenous fistula (dAVF) surgery, externally validated on a microvascular decompression (MVD) surgery, and compared to a fine-tuning approach, AdaptiveSAM, on EndoVis18 dataset. 10/265 frames of dAVF video dataset were used for prompt engineering and fine-tuning and 255/265 for testing. In the MVD dataset, 20/1008 frames were used for training and 988/1008 used for testing. 50 frames were sampled randomly from the Endovis18 training set to train our architecture and it was tested across 1000 frames. Dice Similarity Coefficient (DSC) was calculated across all experiments.

Results: With minimum training, our framework performs accurate segmentation in dAVF and MVD surgical datasets with DSC 0.51(SD=0.05), 0.64(0.06), respectively. Despite being trained on fewer frames of EndoVis18, our architecture was superior (DSC=0.75) compared to AdaptiveSAM (0.69).

Conclusion : Our novel CV framework for a neurosurgical SMI successfully leverages the zero-shot segmentation of SAM and increased accuracy prompted by the AdaptIS network. It is the most sample-efficient and accurate method up-to-date of utilizing SAM in the surgical field and a first in neurosurgery.