[Paper] Constructive Circuit Amplification: Improving Math Reasoning in LLMs via Targeted Sub-Network Updates

Source: arXiv - 2512.16914v1

Overview

A new study shows that you don’t have to retrain an entire large language model (LLM) to make it better at math. By pinpointing and strengthening the tiny “circuits” inside the model that actually do the arithmetic reasoning, the authors boost performance by up to +11.4 % while touching less than 2 % of the model’s parameters. This “Constructive Circuit Amplification” (CCA) opens the door to surgical, low‑cost upgrades for specific capabilities.

Key Contributions

• Circuit‑level intervention: Introduces a systematic way to locate the exact sub‑network (tokens + weights) that drives a target skill (here, mathematical reasoning).
• Sparse fine‑tuning: Updates only ~1.5 % of model components, dramatically reducing compute and data requirements compared with full‑model fine‑tuning.
• Empirical gains: Demonstrates consistent accuracy improvements of +5–11 % on benchmark math tasks across several model sizes (e.g., LLaMA‑7B, LLaMA‑13B).
• Minimal side‑effects: Shows that the targeted updates leave unrelated abilities (MMLU, TriviaQA, TruthfulQA) essentially unchanged.
• Open‑source tooling: Releases the code for token‑trace extraction, circuit identification, and selective weight updating, enabling reproducibility and community extensions.

Methodology

1. Collect reasoning traces – Run the LLM on a set of math problems and record the intermediate token activations (the “thought process”).
2. Identify pivotal tokens – Use attribution techniques (e.g., gradient‑based saliency, attention rollout) to rank which tokens most influence the final answer.
3. Map tokens to model components – Trace the high‑impact tokens back to the underlying weight matrices and attention heads that process them.
4. Select a sparse sub‑network – Keep only the top‑k components (≈1.5 % of total parameters) that are most responsible for the math reasoning path.
5. Targeted fine‑tuning – Fine‑tune this sub‑network on a modest math dataset while freezing the rest of the model.
6. Evaluation – Test the amplified model on standard math benchmarks and on unrelated tasks to measure any collateral damage.

The whole pipeline is fully automated, requiring no manual inspection of the model internals.

Results & Findings

Other abilities (MMLU, TriviaQA, TruthfulQA) changed by less than 0.3 %, confirming that the updates are highly localized.

Key takeaways

• Sparse updates are enough – strengthening the right few heads and MLP rows yields large gains.
• Cross‑model consistency – the same CCA pipeline works on different model scales without re‑engineering.
• Efficiency – training time drops by roughly an order of magnitude compared with full‑model fine‑tuning.

Practical Implications

• Cost‑effective capability upgrades – Companies can roll out math‑or‑domain‑specific improvements without the massive GPU budget of full fine‑tuning.
• Rapid A/B testing – Since only a tiny weight slice changes, you can spin up multiple “skill‑enhanced” variants and compare them in production.
• Safety & alignment – Targeted amplification limits the risk of unintentionally degrading unrelated behavior, a common concern with broad fine‑tuning.
• Modular model design – The work suggests a future where LLMs are built as collections of interchangeable circuits that can be swapped or upgraded on demand (e.g., a “math module” that can be hot‑patched).
• Tooling for developers – The released code can be integrated into existing fine‑tuning pipelines (e.g., Hugging Face Trainer) to add a “circuit‑amplify” step.

Limitations & Future Work

• Scope limited to math – While the method works well for arithmetic reasoning, its effectiveness on more abstract or multimodal tasks remains untested.
• Reliance on trace quality – The quality of the identified circuit depends on the fidelity of the reasoning trace; noisy or ambiguous traces could lead to sub‑optimal sub‑networks.
• Static sparsity budget – The paper uses a fixed ~1.5 % update budget; adaptive budgets that balance performance gain vs. parameter budget could yield better trade‑offs.
• Long‑term stability – The authors note a slight drift in performance after many downstream fine‑tuning steps, suggesting that periodic re‑amplification may be needed.

Future directions include extending CCA to other reasoning domains (code generation, commonsense), automating the sparsity‑budget selection, and exploring “circuit‑level” ensembling where multiple specialized subnetworks are composed at inference time.

Authors

• Nikhil Prakash
• Donghao Ren
• Dominik Moritz
• Yannick Assogba

Paper Information

• arXiv ID: 2512.16914v1
• Categories: cs.CL
• Published: December 18, 2025
• PDF: Download PDF