[Paper] M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding

Source: arXiv - 2601.08758v1

Overview

The paper introduces M3CoTBench, a new benchmark that evaluates how well multimodal large language models (MLLMs) can perform chain‑of‑thought (CoT) reasoning on medical images. By focusing on the reasoning steps—not just the final diagnosis—the authors aim to push AI toward the transparent, step‑by‑step thinking that clinicians use every day.

Key Contributions

• First CoT‑focused benchmark for medical imaging – evaluates correctness, efficiency, impact, and consistency of the reasoning process.
• Broad dataset covering 24 examination types (e.g., X‑ray, CT, MRI) and 13 tasks ranging from simple classification to multi‑step diagnostic reasoning.
• Multi‑level difficulty design that tests models on easy, medium, and hard clinical scenarios.
• Comprehensive evaluation suite with new metrics tailored to clinical reasoning (e.g., reasoning impact on final decision).
• Empirical analysis of several state‑of‑the‑art MLLMs, exposing current gaps in transparent medical reasoning.

Methodology

1. Data Curation – The authors collected publicly available medical imaging cases and annotated them with ground‑truth reasoning chains written by radiologists. Each case includes the image, a clinical question, the step‑by‑step reasoning, and the final answer.
2. Task Design – Thirteen tasks are defined (e.g., “Identify the abnormality”, “Explain why the abnormality is present”, “Suggest next imaging”). Tasks are grouped into three difficulty tiers based on the number of reasoning hops required.
3. Benchmark Construction – For each case, the benchmark records four evaluation dimensions:
   • Correctness – Does the final answer match the expert label?
   • Efficiency – How many reasoning steps does the model generate versus the gold standard?
   • Impact – Does each reasoning step meaningfully contribute to the final decision?
   • Consistency – Are the reasoning steps logically coherent and free of contradictions?
4. Model Evaluation – Several open‑source and commercial MLLMs (e.g., GPT‑4V, LLaVA‑Med, Med-Flamingo) are prompted to produce CoT outputs. Their responses are automatically scored against the benchmark metrics using a mix of lexical matching, semantic similarity (via embedding models), and rule‑based consistency checks.

Results & Findings

• Overall performance is modest: Even the strongest model (GPT‑4V) achieves ~58 % correctness on the hardest tier, far below radiologist levels.
• Reasoning quality lags behind answer accuracy: Models often produce plausible final diagnoses but generate incoherent or redundant reasoning steps, leading to low impact and consistency scores.
• Efficiency trade‑offs: Larger models tend to write longer chains, improving correctness slightly but hurting efficiency (more steps than necessary).
• Task‑specific gaps: Tasks requiring comparative reasoning (e.g., “differentiate between pneumonia and atelectasis”) show the biggest drop in impact scores, indicating current MLLMs struggle with nuanced visual distinctions.

Practical Implications

• Debuggable AI assistants – By exposing the reasoning chain, developers can pinpoint where a model went wrong (e.g., mis‑identified anatomical region) and apply targeted fine‑tuning or rule‑based post‑processing.
• Regulatory readiness – Transparent CoT outputs align with emerging AI‑in‑healthcare guidelines that demand explainability, making it easier to build compliant diagnostic support tools.
• Human‑in‑the‑loop workflows – Clinicians can review the AI’s step‑by‑step logic, accept or reject individual reasoning hops, and thus retain control while still benefiting from AI‑driven suggestions.
• Benchmark‑driven development – M3CoTBench gives product teams a concrete yardstick to measure improvements in both accuracy and interpretability, encouraging the next generation of “explain‑first” MLLMs.

Limitations & Future Work

• Dataset scope – While diverse, the benchmark still relies on publicly available images; rare diseases and non‑English clinical notes are under‑represented.
• Annotation bias – Reasoning chains are authored by a limited group of radiologists, which may not capture the full variability of clinical thought processes.
• Metric automation – Some impact and consistency assessments require manual verification; future work could refine fully automated, clinically validated scoring.
• Model generalization – The study focuses on a handful of MLLMs; extending the benchmark to emerging open‑source models and domain‑specific fine‑tuned versions will be essential.

Bottom line: M3CoTBench shines a light on the “how” behind AI diagnoses, pushing the field toward models that not only get the answer right but can also explain their reasoning in a clinically meaningful way. For developers building AI‑powered health tools, it offers a practical roadmap to more trustworthy, transparent, and regulator‑friendly systems.

Authors

• Juntao Jiang
• Jiangning Zhang
• Yali Bi
• Jinsheng Bai
• Weixuan Liu
• Weiwei Jin
• Zhucun Xue
• Yong Liu
• Xiaobin Hu
• Shuicheng Yan

Paper Information

• arXiv ID: 2601.08758v1
• Categories: eess.IV, cs.CV
• Published: January 13, 2026
• PDF: Download PDF