[Paper] Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models

Source: arXiv - 2606.08571v1

Overview

Large language models frequently fail in a characteristic way: rather than acknowledging ignorance, they produce fluent but incorrect answers to questions that lie beyond their knowledge boundaries. We introduce \textbf{Structured Ignorance Certificates} (SICs), a JSON-formatted output schema that demands a model explicitly name the missing domain intersection, enumerate required concepts, and propose a productive retrieval query rather than hallucinating an answer. To train models to produce high-quality SICs we construct a 7,347-sample \emph{Unknown-Unknown} (UU) dataset by prompting Qwen3-14B to stitch together questions from seven domains (physics, biology, engineering, CS, economics, medical, legal) into novel cross-domain queries that no single-domain expert could answer. We fine-tune a 14B-parameter model with Group Relative Policy Optimization (GRPO) using a composite reward that combines retrieval utility, concept specificity, and output-format validity. A paraphrase-divergence probe trained on model responses confirms that SIC-tuned outputs systematically exhibit higher unknown-unknown probability scores. Evaluation on 735 held-out UU questions achieves a 99.46% JSON validity rate, a mean Certificate Specificity Score of 0.967, and a 3.6% ROUGE-L improvement over the base model on retrieval-grounded generation — demonstrating that explicit epistemic structuring is a learnable and measurable capability.

Key Contributions

This paper presents research in the following areas:

• cs.CL
• cs.AI
• cs.LG

Methodology

Please refer to the full paper for detailed methodology.

Practical Implications

This research contributes to the advancement of cs.CL.

Authors

• Subramanyam Sahoo

Paper Information

• arXiv ID: 2606.08571v1
• Categories: cs.CL, cs.AI, cs.LG
• Published: June 7, 2026
• PDF: Download PDF