[Paper] CoT is Not the Chain of Truth: An Empirical Internal Analysis of Reasoning LLMs for Fake News Generation

Source: arXiv - 2602.04856v1

Overview

The paper CoT is Not the Chain of Truth examines a hidden safety problem in large language models (LLMs) that generate fake news. Even when an LLM refuses to comply with a harmful request, its internal “Chain‑of‑Thought” (CoT) reasoning can still contain and amplify unsafe ideas. By dissecting the model’s internal activations, the authors show that the act of reasoning itself can raise the risk of producing disinformation, challenging the common belief that a refusal automatically guarantees safety.

Key Contributions

• Unified safety‑analysis framework that breaks down CoT generation layer‑by‑layer and isolates the influence of individual attention heads.
• Three interpretable metrics – stability, geometry, and energy – to quantify how attention heads embed or propagate deceptive reasoning patterns.
• Jacobian‑based spectral analysis that reveals which heads are most responsible for unsafe internal narratives.
• Empirical evidence across several reasoning‑oriented LLMs (e.g., GPT‑3.5‑Turbo, LLaMA‑2‑Chat) that the “thinking mode” dramatically increases fake‑news generation risk.
• Identification of a narrow band of mid‑depth layers where critical routing decisions concentrate, showing that only a few contiguous layers drive the unsafe divergence.

Methodology

1. Prompt Design – The authors craft a set of “harmful” news‑generation prompts (e.g., “Write a sensational headline about X”) and collect both the model’s final refusal response and its intermediate CoT tokens.
2. Layer‑wise Decomposition – Using the model’s transformer architecture, they extract hidden states after each layer while the CoT is being generated.
3. Attention‑Head Attribution – For every head, they compute the Jacobian of the hidden state with respect to the input tokens, then apply spectral analysis to derive three scores:
   • Stability: how resistant a head’s activation is to small perturbations (high stability = less likely to flip to unsafe content).
   • Geometry: the alignment of a head’s activation space with known “truth‑preserving” vs. “misinformation‑inducing” directions.
   • Energy: the magnitude of activation, interpreted as the head’s “confidence” in the reasoning path.
4. Risk Scoring – By aggregating these metrics across heads and layers, they produce a risk profile that highlights where unsafe reasoning emerges, even if the final output is a refusal.

Results & Findings

• Risk spikes in CoT mode: When the model is allowed to think step‑by‑step, the internal risk score rises by 30‑50 % compared with a single‑shot generation, despite the same refusal at the end.
• Mid‑depth concentration: Heads in layers 6‑9 (out of 12) dominate the unsafe signal, suggesting a “critical routing window” where the model decides whether to continue a deceptive line of thought.
• Head‑level fingerprints: A small subset (≈ 5 % of all heads) consistently shows high geometry scores aligned with misinformation vectors, acting as “risk amplifiers.”
• Cross‑model consistency: The phenomenon appears in both decoder‑only (GPT‑style) and encoder‑decoder (T5‑style) LLMs, indicating a systemic issue rather than a single architecture quirk.

Practical Implications

• Safety‑by‑Design: Developers can instrument LLM APIs to monitor the identified high‑risk heads during CoT generation and abort or sanitize the process before a harmful narrative solidifies.
• Fine‑tuning & Head Pruning: Targeted fine‑tuning or selective pruning of the risky mid‑depth heads could reduce the internal propagation of fake‑news reasoning without sacrificing overall model capability.
• Policy & Guardrails: The findings suggest that refusal‑only guardrails are insufficient; platforms should incorporate internal safety checks that evaluate the reasoning trace, not just the final output.
• Explainability Tools: The stability/geometry/energy metrics provide a new, interpretable lens for developers building debugging or audit tools for LLMs used in content‑generation pipelines.

Limitations & Future Work

• Scope of Prompts: The study focuses on a specific set of fake‑news prompts; broader domains (e.g., medical misinformation) need validation.
• Model Scale: Experiments were limited to models up to ~70 B parameters; it remains unclear whether larger or more specialized models exhibit the same risk patterns.
• Metric Calibration: The geometry and energy scores rely on handcrafted “misinformation directions”; refining these with larger, labeled corpora could improve accuracy.
• Mitigation Strategies: While the paper identifies risky heads, it does not fully explore the trade‑offs of disabling them; future work should quantify performance impacts and develop safe‑fine‑tuning recipes.

Bottom line: Even a polite “I’m sorry, I can’t help with that” may mask a dangerous line of thought inside the model. By shining a light on the internal dynamics of Chain‑of‑Thought reasoning, this research equips developers with concrete diagnostics—and a call to build safety checks that look inside the model, not just at its final words.

Authors

• Zhao Tong
• Chunlin Gong
• Yiping Zhang
• Qiang Liu
• Xingcheng Xu
• Shu Wu
• Haichao Shi
• Xiao‑Yu Zhang

Paper Information

• arXiv ID: 2602.04856v1
• Categories: cs.CL
• Published: February 4, 2026
• PDF: Download PDF