[Paper] PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training
Source: arXiv - 2606.06470v1
Overview
We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular‑value spectrum of weight matrices via low‑degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama‑1B pre‑training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum‑control principle by proving that uniformly bounding each layer’s singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at https://github.com/Empath-aln/PC-layer.
Key Contributions
- cs.LG
- cs.AI
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.LG.
Authors
- Senmiao Wang
- Tiantian Fang
- Haoran Zhang
- Yushun Zhang
- Kunxiang Zhao
- Alex Schwing
- Ruoyu Sun
Paper Information
- arXiv ID: 2606.06470v1
- Categories: cs.LG, cs.AI
- Published: June 4, 2026
- PDF: Download PDF