[Paper] Hierarchical Orthogonal Residual Spread for Precise Massive Editing in Large Language Models

Source: arXiv - 2601.11441v1

Overview

The paper introduces HORSE (Hierarchical Orthogonal Residual Spread), a new technique for editing large language models (LLMs) at scale. By re‑thinking how knowledge updates are propagated through the model’s internal information matrix, HORSE achieves precise, massive edits while cutting down on noisy gradients and computational overhead—an important step toward safer, more controllable LLM deployments.

Key Contributions

• Hierarchical Orthogonal Residual Spread (HORSE): A novel editing framework that isolates the residual (the “new” knowledge) and spreads it orthogonally across model layers, reducing interference with existing parameters.
• Theoretical grounding: Formal comparison with popular editing baselines (e.g., MEND, MEMIT, FT‑LM) showing HORSE’s superior stability and lower gradient noise.
• Scalable experiments: Validation on two benchmark datasets (e.g., CounterFact and WikiEdit) across multiple LLM sizes (7B‑30B parameters), demonstrating consistent performance in massive editing scenarios.
• Open‑source implementation: Full code released, enabling reproducibility and rapid adoption by the community.

Methodology

1. Information Matrix Decomposition – The authors view a model’s weights as an information matrix that encodes both old and new knowledge. Instead of blending them (as prior work does), HORSE extracts the residual component representing the desired edit.
2. Hierarchical Orthogonal Projection – The residual is projected onto orthogonal subspaces at each layer, ensuring that updates do not overlap with directions already used for existing knowledge. This hierarchical treatment respects the depth‑wise structure of transformers.
3. Residual Spread – The orthogonal residuals are then “spread” down the network using a lightweight linear transformation, avoiding costly second‑order gradient computations typical of methods like MEND.
4. Training‑Free Edit Application – Because the update rule is closed‑form, applying an edit requires only a single forward‑backward pass on a small set of exemplars, making the process fast enough for on‑the‑fly model adjustments.

The overall pipeline can be summarized as: (input query → compute residual → orthogonal projection per layer → spread → apply weight delta).

Results & Findings

• Higher precision: HORSE consistently outperforms MEMIT and MEND by 3–5 % on the edited‑fact accuracy metric.
• Reduced forgetting: The orthogonal design keeps the impact on unrelated knowledge low, preserving >94 % of original performance across all tested models.
• Speed & memory: Editing time drops from ~30 seconds per 100 edits (MEND) to <5 seconds, and memory usage is cut by ~40 % because no second‑order Hessian approximations are stored.

The authors also provide a theoretical proof that orthogonal residual spreading minimizes the norm of the gradient noise term, which aligns with the empirical stability observed.

Practical Implications

• Safety patches on the fly: Companies can deploy quick “security patches” to LLMs (e.g., removing harmful misinformation) without retraining the entire model.
• Customizable enterprise bots: Business‑specific facts (product specs, policy updates) can be injected into a shared LLM instance, ensuring each client sees the correct information while the base model stays unchanged.
• Cost‑effective model maintenance: Since HORSE avoids expensive second‑order calculations, large‑scale model operators can edit millions of facts with modest GPU budgets.
• Regulatory compliance: Rapid removal or alteration of prohibited content becomes feasible, helping organizations meet emerging AI governance requirements.

Limitations & Future Work

• Edit granularity: HORSE excels at factual edits but its effectiveness on more nuanced behavior changes (e.g., style or ethical reasoning) remains untested.
• Scalability to >100 B‑parameter models: Experiments stop at 30 B parameters; the authors note that memory‑efficient orthogonal projections for extremely large models need further engineering.
• Dataset bias: The benchmarks focus on English factual statements; extending evaluation to multilingual or domain‑specific corpora is a planned next step.

Authors

• Xiaojie Gu
• Guangxu Chen
• Yuheng Yang
• Jingxin Han
• Andi Zhang

Paper Information

• arXiv ID: 2601.11441v1
• Categories: cs.CL, cs.AI, cs.LG
• Published: January 16, 2026
• PDF: Download PDF