[Paper] Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach

Source: arXiv - 2511.23335v1

Overview

This paper tackles a long‑standing pain point in modern language models: they can generate fluent text, but we rarely know why they chose the words they did. By introducing a “structured knowledge hunter” that works with both high‑level entities and low‑level knowledge triples, the authors make the generation process transparent while still keeping the output high‑quality. Their approach works across tasks—table‑to‑text and dialogue—showing it can be a general solution for interpretable, knowledge‑enhanced generation.

Key Contributions

• Task‑agnostic structured knowledge hunter that discovers relevant entities and triples without relying on domain‑specific retrievers.
• Local‑global interaction scheme for learning robust representations of hierarchical knowledge (entity ↔ triple).
• Hierarchical transformer‑based pointer network that selects the most pertinent knowledge pieces during generation.
• Unified framework that couples the knowledge hunter with any pretrained language model, boosting both faithfulness and interpretability.
• State‑of‑the‑art results on two benchmarks (RotoWireFG for table‑to‑text, KdConv for dialogue) surpassing existing methods and the underlying language models.

Methodology

1. Structured Knowledge Representation

   • Knowledge is organized in two tiers:
     • Entities (e.g., “LeBron James”) – high‑level concepts.
     • Triples (e.g., “(LeBron James, scored, 30 points)”) – low‑level factual statements.
   • A local encoder captures the semantics of each triple, while a global encoder aggregates information across all triples belonging to the same entity.

2. Local‑Global Interaction

   • The model iteratively exchanges signals between the local and global encoders, allowing each triple to be aware of its entity context and vice‑versa. This yields richer, context‑aware embeddings for both levels.

3. Hierarchical Pointer Network

   • Built on a transformer backbone, the pointer network first decides which entity should be referenced, then which triple under that entity to pull.
   • The selected knowledge pieces are fed as additional conditioning tokens to the downstream language model (e.g., GPT‑2/3), guiding the generation.

4. Training & Inference

   • Supervised learning with cross‑entropy loss for both the pointer selections and the final text generation.
   • During inference, the model outputs not only the generated sentence but also the trace of selected entities/triples, giving a clear explanation of the factual basis for each token.

Results & Findings

• Interpretability: Human evaluators could follow the generated knowledge trace and correctly identify the factual source for >90 % of the content, a stark improvement over opaque baselines.
• Generalization: The same hunter architecture was reused across both datasets without any task‑specific tuning, confirming its task‑agnostic claim.

Practical Implications

• Debuggable AI assistants – Developers can surface the exact knowledge triples a chatbot used to answer a user, making compliance audits and error analysis far simpler.
• Fact‑checking pipelines – The explicit knowledge trace can be fed into downstream verification tools, reducing the risk of hallucinations in generated reports or summaries.
• Plug‑and‑play augmentation – Since the hunter sits on top of any pretrained LM, teams can upgrade from a vanilla model to an interpretable version with minimal code changes.
• Domain‑agnostic data enrichment – Organizations can feed their own knowledge graphs (product catalogs, medical ontologies, etc.) into the hunter and instantly gain transparent generation without building custom retrievers.

Limitations & Future Work

• Knowledge coverage: The approach assumes the required facts are already present in a structured graph; missing triples lead to gaps that the model cannot fill.
• Scalability: Pointer selection over very large knowledge bases can become computationally heavy; the authors suggest hierarchical pruning as a next step.
• Evaluation breadth: Experiments are limited to two benchmarks; broader testing on open‑domain QA or long‑form generation would strengthen claims.
• User studies: While interpretability was measured with expert annotators, real‑world user studies (e.g., in customer‑support settings) are left for future research.

Overall, the paper presents a compelling blueprint for making language‑model generation both faithful and explainable, a combination that many production teams are actively seeking.

Authors

• Shuqi Liu
• Han Wu
• Guanzhi Deng
• Jianshu Chen
• Xiaoyang Wang
• Linqi Song

Paper Information

• arXiv ID: 2511.23335v1
• Categories: cs.CL, cs.AI
• Published: November 28, 2025
• PDF: Download PDF