[Paper] Memory Bank Compression for Continual Adaptation of Large Language Models

Source: arXiv - 2601.00756v1

Overview

The paper “Memory Bank Compression for Continual Adaptation of Large Language Models” tackles a pressing problem: keeping massive language models up‑to‑date as new data streams in, without blowing up memory or erasing what the model already knows. The authors introduce MBC, a technique that compresses the external memory bank used by continual‑learning LLMs, enabling efficient online updates while preserving prior knowledge.

Key Contributions

• Memory‑Bank Compression (MBC): A codebook‑based optimization that shrinks the external memory to a fraction of its original size (≈0.3 % of the baseline).
• Online Resetting Mechanism: Prevents the learned codebook from collapsing during streaming updates, ensuring stable adaptation.
• Key‑Value Low‑Rank Adaptation (KV‑LoRA): Integrates compressed memory vectors into the LLM’s attention layers with minimal extra parameters.
• Empirical Validation: Demonstrates that MBC retains high accuracy on benchmark QA tasks while drastically reducing memory footprint.
• Open‑Source Release: Full implementation and scripts are made publicly available, encouraging reproducibility and downstream use.

Methodology

1. Memory Bank as a Retrieval Store – In many continual‑learning setups, an LLM is paired with an external “memory bank” that holds embeddings of past examples. During inference, the model retrieves the most relevant entries to augment its predictions.
2. Codebook Optimization – Instead of storing every raw embedding, MBC learns a codebook of a limited set of prototype vectors. Each new memory entry is quantized to the nearest prototype, dramatically reducing storage.
3. Online Resetting – As new data streams in, the distribution of embeddings can shift, risking that many prototypes become unused (codebook collapse). The authors periodically re‑initialize under‑utilized prototypes based on current data statistics, keeping the codebook expressive.
4. KV‑LoRA Integration – The compressed memory vectors are injected into the LLM’s attention mechanism via low‑rank updates to the key and value projection matrices. This adds only a tiny number of trainable parameters, preserving the original model’s efficiency.
5. Training Loop – The system performs online updates: each incoming batch triggers (a) quantization into the codebook, (b) a forward pass with KV‑LoRA‑augmented attention, and (c) a lightweight gradient step on the LoRA parameters and the codebook vectors.

Results & Findings

• Compression Ratio: MBC achieves a ~300× reduction in memory usage compared with the strongest existing method.
• Retention Accuracy: The drop in QA performance is less than 0.5 %, indicating that the compressed representations still capture the essential information.
• Computation: Because only LoRA parameters are updated, each online step incurs ≈2–3× less GPU time than full fine‑tuning.
• Stability: The online resetting mechanism eliminates catastrophic degradation of the codebook, as shown by smooth loss curves across long streaming runs.

Practical Implications

• Edge & On‑Device AI: Devices with limited storage (e.g., smartphones, IoT gateways) can now host a “memory‑augmented” LLM that stays current without needing to download massive update packages.
• Enterprise Knowledge Bases: Companies can continuously feed internal documents into a large language model while keeping the auxiliary memory lightweight, enabling up‑to‑date chat‑bots or search assistants.
• Cost‑Effective Model Maintenance: Reducing memory and compute overhead translates directly into lower cloud‑hosting bills for services that rely on continual learning (e.g., personalized recommendation engines).
• Rapid Prototyping: Developers can experiment with streaming data pipelines (news feeds, logs) and see immediate model improvements without the risk of catastrophic forgetting.
• Compatibility: Since MBC works on top of any transformer‑based LLM and only adds LoRA‑style adapters, it can be dropped into existing codebases with minimal refactoring.

Limitations & Future Work

• Codebook Size Selection: The optimal number of prototypes is dataset‑dependent; the paper relies on a heuristic search, which may be cumbersome for new domains.
• Long‑Term Drift: While the resetting mechanism mitigates collapse, the codebook can still become stale if the data distribution shifts dramatically over months—future work could explore continual codebook growth or hierarchical prototypes.
• Evaluation Scope: Experiments focus on QA benchmarks; applying MBC to generation‑heavy tasks (e.g., dialogue, code synthesis) remains an open question.
• Hardware Specificity: The current implementation assumes GPU‑friendly quantization; adapting the approach to specialized accelerators (TPUs, edge NPUs) may require additional engineering.

Overall, MBC offers a compelling recipe for making continual‑learning LLMs practical at scale, opening the door for more responsive and memory‑efficient AI services.

Authors

• Thomas Katraouras
• Dimitrios Rafailidis

Paper Information

• arXiv ID: 2601.00756v1
• Categories: cs.LG, cs.CL
• Published: January 2, 2026
• PDF: Download PDF