[Paper] FlashOverlap: Minimizing Tail Latency in Communication Overlap for Distributed LLM Training

Source: arXiv - 2604.24013v1

Overview

Training today’s massive language models requires spreading work across many GPUs or other accelerators, but the resulting data‑movement can become a serious bottleneck. FlashOverlap proposes a fresh way to overlap communication and computation that eliminates the “tail latency” problem that plagues existing slice‑based overlap schemes, delivering faster, more efficient distributed LLM training.

Key Contributions

• Flash‑Overlap algorithm – Replaces heavyweight collective ops (reduce‑scatter, all‑gather) with a series of fine‑grained peer‑to‑peer (P2P) transfers that can be interleaved with computation.
• Exact latency‑optimal scheduling – Provides a provably optimal schedule that removes the long‑tail latency observed in prior overlap methods.
• Broad compatibility – Works with pure data‑parallel training as well as tensor‑parallel strategies such as Tensor‑Parallelism (TPSP) and Unified Parallelism (UP).
• Empirical gains – Demonstrates consistent reductions in overall step time, higher Model FLOPS Utilization (MFU), and increased throughput across multiple model sizes and hardware configurations.

Methodology

1. Decompose Collectives – Instead of issuing a single collective call, Flash‑Overlap breaks it into a set of directed P2P messages (e.g., point‑to‑point sends/receives). This gives the runtime fine control over when each piece of data moves.
2. Partitioned Computation – The forward/backward kernels are split into smaller sub‑tasks that operate on the same tensor slices that are being communicated.
3. Latency‑Optimal Scheduler – An analytical model evaluates the dependency graph of P2P transfers and compute fragments, then produces a schedule that maximizes overlap while guaranteeing that no sub‑task waits for the “last” piece of data (the tail).
4. Integration Layer – The authors wrap the scheduler inside popular deep‑learning frameworks (PyTorch + NCCL) so that existing training scripts can switch to Flash‑Overlap with minimal code changes.

The approach is deliberately kept implementation‑friendly: it relies only on standard P2P primitives already exposed by NCCL, MPI, or custom interconnect libraries, and does not require hardware modifications.

Results & Findings

• Tail latency eliminated – The longest‑waiting communication fragment is reduced to near‑zero, flattening the step‑time distribution.
• Higher MFU – More of the GPU’s compute capacity is kept busy, indicating better resource utilization.
• Scalable across parallelism schemes – Gains hold for pure data parallelism as well as mixed tensor‑parallel configurations.

Practical Implications

• Faster model iteration – Teams can cut training time by up to a quarter without adding hardware, accelerating research cycles and product development.
• Cost savings – Reduced step time translates directly into lower cloud‑GPU bills, especially for multi‑node runs where communication dominates the expense.
• Simplified scaling – Because Flash‑Overlap works with existing P2P primitives, it can be adopted on any cluster that already runs NCCL/MPI, making it a drop‑in upgrade for large‑scale LLM pipelines.
• Inference benefits – The same overlap technique can be applied to tensor‑parallel inference, lowering latency for serving massive models in production.

Limitations & Future Work

• Dependency on network topology – The optimal schedule assumes relatively uniform bandwidth; highly heterogeneous interconnects (e.g., mixed Ethernet/InfiniBand) may need custom tuning.
• Kernel partitioning overhead – Splitting large kernels incurs modest bookkeeping cost; for very small models the benefit diminishes.
• Framework integration depth – Current prototypes target PyTorch; extending to TensorFlow or JAX will require additional engineering.

Future directions include automated topology‑aware schedule generation, tighter integration with emerging communication libraries (e.g., NCCL‑3), and exploring adaptive runtime decisions that switch between collective and P2P modes based on real‑time bandwidth measurements.

FlashOverlap shows that rethinking how we interleave communication and computation—down to the level of individual peer‑to‑peer messages—can unlock measurable performance gains for the biggest language models being trained today. For developers managing large‑scale training clusters, the technique offers a practical, hardware‑agnostic lever to shave latency and boost throughput.

Authors

• Rezaul Karim
• Austin Wen
• Wang Zongzuo
• Weiwei Zhang
• Yang Liu
• Walid Ahmed

Paper Information

• arXiv ID: 2604.24013v1
• Categories: cs.LG, cs.CV, cs.DC
• Published: April 27, 2026
• PDF: Download PDF