[Paper] Towards Resiliency in Large Language Model Serving with KevlarFlow

Source: arXiv - 2601.22438v1

Overview

Large Language Model (LLM) serving platforms are increasingly the backbone of AI‑powered products, yet they remain surprisingly fragile: a single hardware glitch in a hyperscale cluster can cascade into multi‑minute service outages. The paper Towards Resiliency in Large Language Model Serving with KevlarFlow proposes a new serving architecture that dramatically speeds up recovery and keeps latency low even when parts of the system fail.

Key Contributions

• KevlarFlow architecture that decouples model‑parallel initialization from request handling, allowing new workers to join without pausing the service.
• Dynamic traffic rerouting that automatically redirects inference requests around failed nodes, preserving throughput.
• Background KV‑cache replication that keeps the token‑level attention cache synchronized across replicas, eliminating costly warm‑up delays after a failure.
• Empirical evaluation showing a 20× reduction in mean‑time‑to‑recovery (MTTR) and up to 574× improvement in 99th‑percentile time‑to‑first‑token (TTFT) compared with leading LLM serving stacks.
• Negligible runtime overhead (≤ 2 % extra latency) when the system operates without failures, proving the approach is production‑ready.

Methodology

1. Decoupled Model Parallelism – Instead of launching a monolithic pipeline that blocks until every GPU shard loads the full model, KevlarFlow spins up each shard independently. A lightweight coordinator tracks which shards are ready and starts routing traffic to them as soon as they become available.
2. Dynamic Traffic Rerouting – A health‑monitoring layer continuously probes each shard. When a failure is detected, the router updates its forwarding table in real time, sending new inference requests to the remaining healthy shards. Existing in‑flight requests are either completed on the surviving shards or gracefully aborted.
3. Background KV‑Cache Replication – The KV (key‑value) cache that stores attention states for each conversation is replicated asynchronously across a standby replica set. If a primary shard crashes, the standby already holds a fresh copy of the cache, so the new shard can resume generation without recomputing the entire context.
4. Evaluation Setup – Experiments were run on a 64‑GPU cluster using popular LLMs (e.g., LLaMA‑13B, Falcon‑40B). Faults were injected by programmatically killing GPU processes or cutting network links, and metrics such as latency, throughput, MTTR, and TTFT were recorded against baseline serving frameworks (vLLM, DeepSpeed‑Inference).

Results & Findings

The numbers show that KevlarFlow not only recovers dramatically quicker but also keeps user‑facing latency low during and after a failure, all while adding almost no extra cost when the system is healthy.

Practical Implications

• Higher SLA compliance – Services that promise sub‑second responses can now survive hardware hiccups without breaching latency SLAs.
• Cost‑effective scaling – Operators can run larger clusters with lower redundancy because KevlarFlow mitigates the impact of individual node failures, reducing the need for over‑provisioned standby pools.
• Developer ergonomics – The decoupled initialization model means engineers can roll out new model versions or add GPU shards without taking the whole service offline.
• Better UX for conversational AI – Faster TTFT translates directly into smoother chat experiences, especially important for real‑time assistants, code‑completion tools, and gaming bots.
• Simplified ops tooling – Since traffic rerouting and cache replication are baked into the serving stack, existing monitoring and orchestration pipelines (Kubernetes, Prometheus) need only minimal custom logic.

Limitations & Future Work

• Cache consistency trade‑offs – The asynchronous KV‑cache replication can, in rare edge cases, serve slightly stale context if a failure occurs mid‑update. The authors suggest exploring stronger consistency protocols.
• Hardware diversity – Experiments focused on homogeneous GPU clusters; extending KevlarFlow to heterogeneous environments (CPU‑only nodes, TPUs) remains an open challenge.
• Model size ceiling – While the approach scales to 40‑B‑parameter models, ultra‑large models (>100 B) may still hit bandwidth bottlenecks during cache sync, prompting research into more efficient compression or delta‑encoding techniques.
• Security considerations – Replicating KV caches across nodes introduces additional attack surface; future work should integrate encryption and access‑control mechanisms.

Authors

• Shangshu Qian
• Kipling Liu
• P. C. Sruthi
• Lin Tan
• Yongle Zhang

Paper Information

• arXiv ID: 2601.22438v1
• Categories: cs.DC, cs.CL, cs.LG
• Published: January 30, 2026
• PDF: Download PDF