[Paper] Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

Source: arXiv - 2601.07820v1

Overview

The paper investigates whether modern vision‑language models can behave like a conversational partner who asks for clarification when they are unsure about a referent. By framing the problem as a reference game—a controlled setting where a speaker describes an object and a listener must identify it—the authors create a measurable test of a model’s ability to recognize its own uncertainty and request additional information.

Key Contributions

• Novel testbed: Introduces reference games as a lightweight, reproducible benchmark for probing uncertainty‑aware behavior in multimodal models.
• Clarification protocol: Defines a simple instruction set that lets models explicitly request clarification instead of guessing.
• Empirical evaluation: Benchmarks three state‑of‑the‑art vision‑language models on both a standard reference‑resolution task and the new “clarify‑when‑uncertain” variant.
• Diagnostic insights: Shows that even on modest, well‑structured tasks, current models often fail to translate internal uncertainty into appropriate clarification requests.
• Open‑source resources: Provides the game data, prompts, and evaluation scripts to encourage community adoption.

Methodology

1. Reference Game Setup – Each round presents an image containing several objects. A textual description (the “speaker”) refers to one target object using attributes (color, shape, location, etc.).
2. Baseline Task – Models receive the description and must output the index of the target object (reference resolution). Accuracy on this task serves as a performance ceiling.
3. Clarification Condition – Models are additionally told: “If you are not confident about which object is meant, ask a clarification question; otherwise, give your answer.” The model can either (a) answer directly or (b) generate a clarification request (e.g., “Do you mean the red cup on the left?”).
4. Uncertainty Detection – No explicit confidence score is required; the model’s internal representation is probed via prompting. The authors treat a generated clarification request as evidence that the model recognized uncertainty.
5. Evaluation Metrics –
   • Resolution Accuracy (baseline vs. clarification condition).
   • Clarification Appropriateness – whether a request is issued when the model would otherwise be wrong, and whether the request is semantically relevant.
   • Precision/Recall of Clarifications – measuring over‑ and under‑asking.

Three publicly available vision‑language models (e.g., BLIP‑2, OFA, and a CLIP‑based encoder‑decoder) are tested under identical prompts.

Results & Findings

• Baseline performance ranged from 78 % to 91 % correct identification, confirming that the games are solvable for current models.
• Clarification behavior was inconsistent:
  • On average, models asked for clarification in only 30‑45 % of the cases where they later made a mistake, indicating low recall of uncertainty.
  • When they did ask, 40‑55 % of the questions were either vague or irrelevant, showing limited precision.
• Model differences: The larger encoder‑decoder (OFA) was slightly better at detecting uncertainty but still failed to request clarification in many high‑risk instances.
• Trade‑off: Forcing a model to ask clarification reduced outright errors (from ~10 % to ~6 % on average) but introduced new failure modes where the model asked unnecessarily, slowing down interaction.

Overall, the study demonstrates that current vision‑language models lack a reliable internal signal that can be surfaced as a human‑like clarification request.

Practical Implications

• Human‑AI Collaboration: In mixed‑initiative systems (e.g., robotic assistants, AR overlays, or visual search tools), the ability to ask “Did you mean…?” can prevent costly mistakes and improve user trust.
• Safety‑Critical Domains: For medical imaging or autonomous inspection, a model that flags uncertainty and seeks clarification could reduce false positives/negatives.
• Prompt Engineering: The paper shows that simple prompting can coax models into uncertainty‑aware behavior, suggesting a low‑cost pathway for developers to add clarification logic without retraining.
• Evaluation Standards: Reference games provide a reproducible benchmark that can be integrated into existing model evaluation pipelines, encouraging developers to consider interaction quality alongside raw accuracy.
• Product Design: UI/UX designers can embed clarification loops (e.g., “Is this the red mug you’re looking for?”) based on model‑generated uncertainty signals, leading to smoother conversational interfaces.

Limitations & Future Work

• Scope of Games: The reference games are visually simple and involve limited vocabularies; performance may differ on richer, real‑world scenes.
• Implicit Confidence: The study relies on prompting to surface uncertainty rather than explicit confidence scores, which may be noisy. Future work could explore calibrated probability outputs.
• Model Size & Training Data: Only three models were examined; larger or instruction‑tuned models (e.g., GPT‑4V) might behave differently.
• User Studies: The paper evaluates model behavior in isolation; real‑world user studies are needed to assess how humans perceive and respond to AI‑generated clarification requests.
• Iterative Clarification: Current experiments allow a single clarification turn. Extending to multi‑step dialogues could reveal richer interaction dynamics.

By highlighting both the promise and the current gaps, the paper sets a clear agenda for building more self‑aware, collaborative AI systems.

Authors

• Manar Ali
• Judith Sieker
• Sina Zarrieß
• Hendrik Buschmeier

Paper Information

• arXiv ID: 2601.07820v1
• Categories: cs.CL
• Published: January 12, 2026
• PDF: Download PDF