[Paper] SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning

Source: arXiv - 2603.06570v1

Overview

The paper introduces SUREON, a new benchmark and vision‑language model (VLM) that teaches AI systems to reason about surgery, not just recognize instruments or anatomy. By mining the spoken explanations from thousands of surgical lecture videos, the authors create a large‑scale question‑answer (QA) dataset that captures surgeons’ intent, safety assessments, and predictions about what will happen next—capabilities that have been missing from existing surgical AI.

Key Contributions

• SUREON dataset: 134.7 K video clips from 170 procedure types, annotated automatically into 206.8 K QA pairs across 12 clinically‑relevant question categories (e.g., safety risk, decision rationale, next‑step forecasting).
• Expert‑validated benchmark: 354 hand‑checked examples that serve as a high‑quality test set for surgical reasoning.
• SureonVLM: A vision‑language model fine‑tuned on the SUREON QA pairs, enabling it to answer complex surgical questions.
• SureonVLM‑R1: An enhanced reasoning model trained with Group Relative Policy Optimization (GRPO) that demonstrates explicit step‑by‑step inference.
• Empirical gains: Both models achieve >84 % accuracy on the SUREON benchmark and outperform larger, general‑domain VLMs on standard surgical perception tasks (instrument detection, phase recognition, etc.).

Methodology

1. Data Harvesting

   • Collected publicly available surgical lecture videos (e.g., from academic conferences, online courses).
   • Used automatic speech‑to‑text to obtain the narrations that surgeons provide while explaining each step.

2. Multi‑Agent Annotation Pipeline

   • Segmentation Agent: splits videos into short, semantically coherent clips.
   • Question Generation Agent: maps narration sentences to one of 12 predefined question templates (e.g., “Why is instrument X used here?”).
   • Answer Extraction Agent: extracts the corresponding answer span from the transcript.
   • This pipeline produces structured QA pairs without manual labeling, while preserving the rich reasoning embedded in the narrations.

3. Model Training

   • SureonVLM: starts from a pre‑trained vision‑language backbone (e.g., CLIP or BLIP) and is fine‑tuned on the SUREON QA pairs using supervised cross‑entropy loss.
   • SureonVLM‑R1: builds on SureonVLM but adds a reinforcement‑learning‑style objective (GRPO) that rewards the model for generating answers that are relatively better within a group of candidate responses, encouraging more explicit reasoning steps.

4. Evaluation

   • Accuracy on the expert‑validated benchmark (354 examples).
   • Transfer tests on existing surgical perception datasets (instrument detection, phase classification) to assess whether reasoning training harms basic perception abilities.

Results & Findings

• Benchmark performance: SureonVLM‑R1 reaches 84.3 % accuracy on the SUREON benchmark, a +22 % relative improvement over the strongest general‑domain VLM (e.g., GPT‑4‑V).
• Reasoning behavior: Qualitative inspection shows the model can articulate why a particular tool is chosen, assess safety (e.g., “Is this step at risk of bleeding?”), and predict the next maneuver.
• Perception transfer: Both SureonVLM variants maintain or slightly improve performance on standard surgical perception tasks, demonstrating that reasoning supervision does not degrade visual understanding.
• Data efficiency: Even with only ~0.2 % of the total QA pairs used for fine‑tuning, the models achieve >80 % benchmark accuracy, suggesting the multi‑agent pipeline yields high‑quality supervision.

Practical Implications

• Intelligent OR assistants: Real‑time AI that can warn surgeons about potential risks (“This clip may cause tissue damage”) or suggest next steps based on the current view.
• Automated surgical education: Interactive tutoring systems that answer trainee questions (“Why is the surgeon switching to a suction device now?”) directly from operative video streams.
• Safety auditing: Post‑operative video analysis that flags moments where the surgeon’s decision deviated from standard safety guidelines, supporting quality‑control workflows.
• Cross‑procedure generalization: Because the model learns reasoning patterns rather than procedure‑specific visual cues, it can be adapted to new surgeries with minimal additional data—valuable for niche specialties.

Limitations & Future Work

• Noisy narration: Automatic speech‑to‑text errors and informal teaching language introduce occasional mis‑alignments between video content and QA pairs.
• Benchmark size: The expert‑validated set is relatively small (354 examples), which may limit statistical confidence for some rare question types.
• Domain scope: Current data focuses on academic lecture videos; intra‑operative recordings (e.g., live surgeries) may have different visual dynamics and less explicit narration.
• Future directions
  • Incorporate multimodal grounding (e.g., aligning instrument tip trajectories with explanations).
  • Expand the benchmark with crowd‑sourced validation to cover more rare procedures.
  • Explore continual learning pipelines that update the model as new surgical videos and annotations become available.

Bottom line: SUREON demonstrates that the “why” behind surgical actions—already embedded in teaching videos—can be harvested at scale to train AI that reasons like a human surgeon. For developers building next‑generation operating‑room assistants or educational tools, this work offers both a rich dataset and a proven modeling recipe to move beyond perception toward true surgical cognition.

Authors

• Alejandra Perez
• Anita Rau
• Lee White
• Busisiwe Mlambo
• Chinedu Nwoye
• Muhammad Abdullah Jamal
• Omid Mohareri

Paper Information

• arXiv ID: 2603.06570v1
• Categories: cs.CV, cs.AI
• Published: March 6, 2026
• PDF: Download PDF