[Paper] Piper: A Programmable Distributed Training System
Source: arXiv - 2606.11169v1
Overview
Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation. Piper allows users to declare a comprehensive distributed training strategy with a small set of model annotations and scheduling directives. Each directive applies a transformation on Piper’s intermediate representation (IR), a unified global training DAG that represents all computation and communication. Using this IR, Piper compiles per-device execution plans and executes them with a distributed runtime agnostic to the strategy. We show that the combined system maintains performance parity on commonly available strategies such as ZeRO, while also enabling additional performance and memory efficiency gains through joint scheduling of compute and communication in composed parallelism strategies such as DeepSeek-V3’s DualPipe.
Key Contributions
This paper presents research in the following areas:
- cs.DC
- cs.AI
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.DC.
Authors
- Megan Frisella
- Shubham Tiwari
- Andy Ruan
- Yi Pan
- Parker Gustafson
- Mat Jacob
- Gilbert Bernstein
- Stephanie Wang
Paper Information
- arXiv ID: 2606.11169v1
- Categories: cs.DC, cs.AI
- Published: June 9, 2026
- PDF: Download PDF