[Paper] CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing

Source: arXiv - 2602.15823v1

Overview

CrispEdit tackles a persistent headache in large language model (LLM) maintenance: how to change a specific behavior (e.g., correcting a factual error) without unintentionally breaking the model’s broader capabilities. By framing editing as a constrained optimization problem and projecting updates onto a “low‑curvature” subspace, the authors deliver a scalable, second‑order method that preserves overall performance while achieving precise edits.

Key Contributions

• Unified constrained‑optimization framework for model editing that explicitly encodes capability preservation as a mathematical constraint.
• Low‑curvature projection using the exact Gauss‑Newton Hessian derived from a Bregman‑divergence formulation, ensuring edits stay in directions that minimally affect unrelated abilities.
• Scalable implementation via Kronecker‑Factored Approximate Curvature (K‑FAC) and a novel matrix‑free projector, making second‑order updates feasible for LLMs with billions of parameters.
• Empirical superiority: on standard editing benchmarks CrispEdit attains higher edit success rates while keeping average capability loss under 1 %, a substantial gain over prior methods.

Methodology

1. Edit Objective – Define a loss that penalizes the model for producing the undesired output on a target input (e.g., “The capital of X is Y”).
2. Capability Constraint – Introduce a separate loss that measures how much the edit harms the model’s general abilities (e.g., language fluency, reasoning). This is expressed as a Bregman divergence, whose quadratic expansion yields the exact Gauss‑Newton Hessian.
3. Constrained Optimization – Solve for the parameter update that minimizes the edit loss subject to keeping the capability loss within a tiny tolerance.
4. Low‑Curvature Projection – Project the raw gradient onto the subspace spanned by eigenvectors with the smallest curvature (i.e., directions where the capability loss changes slowly). This avoids “stepping” into regions that would degrade performance.
5. Scalable Approximation – Compute the required Hessian‑vector products efficiently with K‑FAC, which approximates the full curvature matrix as a Kronecker product of smaller factors. A matrix‑free projector leverages this structure to apply the projection without ever materializing gigantic matrices.

The result is a second‑order edit step that is both precise (thanks to curvature awareness) and practical for models with billions of parameters.

Results & Findings

Key takeaways:

• CrispEdit reliably flips targeted facts while leaving the rest of the model’s behavior virtually untouched.
• The low‑curvature projection is the main driver of the dramatic reduction in capability loss.
• Even when the base model is not fully converged (common in fine‑tuning pipelines), the Gauss‑Newton approximation remains accurate enough to guide safe edits.

Practical Implications

• Rapid Model Maintenance – Teams can push targeted patches (e.g., fixing hallucinations, updating policy‑violating content) without re‑training or extensive validation cycles.
• Compliance & Safety – Regulatory updates (e.g., removing disallowed content) can be enforced with provable bounds on side‑effects, reducing the risk of inadvertent capability regression.
• Continuous Learning – In production systems that ingest user feedback, CrispEdit enables “online” correction of specific errors while preserving the model’s overall skill set.
• Tooling Integration – Because the method relies on standard autograd operations plus K‑FAC, it can be wrapped into existing ML frameworks (PyTorch, JAX) and incorporated into CI pipelines for LLMs.

Limitations & Future Work

• Approximation Quality – K‑FAC is an approximation; in highly non‑convex regions its curvature estimate may be less reliable, potentially limiting edit precision for extremely large or sparsely fine‑tuned models.
• Scope of Edits – CrispEdit excels at local edits (single fact or behavior). Scaling to wholesale architectural changes (e.g., adding new capabilities) remains an open challenge.
• Computational Overhead – Although far cheaper than full Hessian methods, the K‑FAC bookkeeping still adds non‑trivial runtime and memory cost compared to first‑order editors.
• Future Directions – The authors suggest exploring richer constraint families (e.g., multi‑task capability bundles), adaptive curvature thresholds, and tighter integration with reinforcement‑learning‑based safety layers.

---

Bottom line: CrispEdit offers a mathematically grounded, production‑ready recipe for “surgical” LLM updates that keep the rest of the model healthy—a compelling step toward truly maintainable AI systems.

Authors

• Zarif Ikram
• Arad Firouzkouhi
• Stephen Tu
• Mahdi Soltanolkotabi
• Paria Rashidinejad

Paper Information

• arXiv ID: 2602.15823v1
• Categories: cs.LG, cs.AI
• Published: February 17, 2026
• PDF: Download PDF