[Paper] InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration

Source: arXiv - 2512.02981v1

Overview

The paper “InEx: Hallucination Mitigation via Introspection and Cross‑Modal Multi‑Agent Collaboration” tackles one of the most stubborn problems in large language models (LLMs): hallucination—the generation of statements that sound plausible but are factually wrong. By borrowing a human‑like decision‑making process—first introspecting, then seeking external verification—the authors introduce a training‑free, multi‑agent framework that dramatically reduces hallucinations in multimodal LLMs (MLLMs).

Key Contributions

• InEx framework: A novel, plug‑and‑play system that combines internal introspection with cross‑modal, multi‑agent verification, requiring no extra model training.
• Entropy‑based uncertainty estimator: Quantifies how unsure the decision agent is, triggering deeper introspection when needed.
• Three‑agent collaboration:
  1. Decision agent – produces the initial answer.
  2. Editing agent – critiques and rewrites the answer using visual/textual cues.
  3. Self‑reflection agents – run iterative checks and refine the response.
• Empirical gains: Consistent 4 %–27 % improvements over strong baselines on both general QA and dedicated hallucination benchmarks.
• Robustness: Demonstrates stable performance across diverse prompts, modalities, and noise levels.

Methodology

1. Introspective Reasoning – When the decision agent generates an answer, InEx first measures its entropy (a statistical proxy for uncertainty). High entropy triggers the agent to re‑evaluate its reasoning path before committing to a final output.
2. Cross‑Modal Collaboration – The initial answer is handed to an editing agent that can see the accompanying image (or other modality) and the text. It checks for mismatches (e.g., “the cat is white” vs. a clearly black cat) and proposes edits.
3. Self‑Reflection Loop – One or more reflection agents run a lightweight verification pass: they ask “Does this claim follow from the visual evidence?” and either accept, request another edit, or flag the answer as uncertain. This loop repeats until the entropy drops below a preset threshold or a maximum number of iterations is reached.
4. Training‑Free Design – All agents are standard LLMs (or vision‑language models) used off‑the‑shelf; the framework orchestrates them at inference time, avoiding costly fine‑tuning.

Results & Findings

• Benchmark performance: On the MMQA and VQA‑Hallucination suites, InEx achieved up to 27 % higher factual accuracy than baseline MLLMs.
• Generalization: Even when tested on unseen domains (e.g., medical images, technical diagrams), the framework retained a +10 % boost in correctness.
• Ablation studies: Removing the entropy‑based introspection cut performance by ~6 %; dropping the editing agent caused a 12 % drop, confirming each component’s importance.
• Speed trade‑off: The multi‑agent loop adds ~0.8× latency compared to a single‑pass model, but remains within interactive response times (<2 seconds for typical queries).

Practical Implications

• Safer AI assistants: Developers can embed InEx into chatbots, virtual agents, or customer‑support tools that need to reference images (e.g., product manuals) without risking misinformation.
• Low‑cost reliability upgrade: Because InEx works at inference time, existing LLM deployments can be upgraded without retraining, saving compute budgets.
• Regulatory compliance: Industries like healthcare or finance, where hallucinations can have legal consequences, can use InEx to meet higher factual‑accuracy standards.
• Tooling for developers: The authors release a lightweight API that lets engineers compose the three agents with custom prompts, making it easy to adapt the framework to domain‑specific knowledge bases.

Limitations & Future Work

• Latency overhead: The iterative verification loop, while modest, may be prohibitive for ultra‑low‑latency applications (e.g., real‑time gaming).
• Dependency on modality quality: If the visual input is noisy or ambiguous, the editing agent can still propagate errors.
• Scalability of agents: The current design assumes three agents; extending to more complex tasks may require smarter orchestration or dynamic agent selection.
• Future directions: The authors suggest exploring learned policies for when to stop the introspection loop, integrating external knowledge graphs for deeper fact‑checking, and applying the paradigm to purely textual hallucination mitigation.

Authors

• Zhongyu Yang
• Yingfang Yuan
• Xuanming Jiang
• Baoyi An
• Wei Pang

Paper Information

• arXiv ID: 2512.02981v1
• Categories: cs.CV
• Published: December 2, 2025
• PDF: Download PDF