[Paper] FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
Source: arXiv - 2606.08476v1
Overview
Context parallelism (CP) is essential for training large-scale, long-context language models, as it partitions sequences to reduce memory overhead. However, existing CP methods suffer from workload imbalance, inefficient kernels, and redundant communication due to static sequence sharding and key-value (KV) tensor communication. We present FlashCP, a load-balanced and communication-efficient framework for CP training. FlashCP introduces a sharding-aware communication mechanism to eliminate redundant KV communication and proposes a novel Whole-Doc sharding strategy that maximizes communication savings while maintaining balanced workloads. To efficiently combine Whole-Doc and Per-Doc sharding, FlashCP further designs a heuristic algorithm to search for near-optimal sharding plans. Extensive experiments show that FlashCP achieves up to 1.63x speedup over state-of-the-art CP frameworks across diverse datasets.
Key Contributions
This paper presents research in the following areas:
- cs.DC
- cs.AI
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.DC.
Authors
- Zheng Wang
- Eric Liu
- Linan Jiang
- Zhongkai Yu
- Zaifeng Pan
- Yue Guan
- Yuke Wang
- Yufei Ding
Paper Information
- arXiv ID: 2606.08476v1
- Categories: cs.DC, cs.AI
- Published: June 7, 2026
- PDF: Download PDF