[Paper] DHP: Efficient Scaling of MLLM Training with Dynamic Hybrid Parallelism

Source: arXiv - 2602.21788v1

Overview

The paper introduces Dynamic Hybrid Parallelism (DHP), a new way to train multimodal large language models (MLLMs) that can handle long contexts and wildly varying data shapes. By continuously reshaping communication groups and parallelism degrees on‑the‑fly, DHP keeps GPUs/NPUs busy even when the training data is highly heterogeneous, delivering up to a 36 % speed boost over state‑of‑the‑art frameworks such as Megatron‑LM and DeepSpeed.

Key Contributions

• Adaptive parallelism engine that re‑configures both data‑parallel and model‑parallel groups each training step, eliminating static‑strategy bottlenecks.
• Generalized non‑power‑of‑two parallelism support, allowing arbitrary numbers of devices to be used efficiently without padding or waste.
• Polynomial‑time strategy optimizer that computes near‑optimal parallelism configurations in only a few milliseconds per batch.
• Empirical validation on large NPU clusters showing up to 1.36× higher throughput and near‑linear scaling despite extreme data heterogeneity.
• Open‑source reference implementation compatible with existing Megatron‑LM/DeepSpeed pipelines, easing adoption for practitioners.

Methodology

1. Profiling per‑batch characteristics – Before each forward/backward pass, DHP gathers lightweight statistics (e.g., token length, image resolution, modality mix) that indicate the computational load of the current batch.
2. Dynamic group formation – Using these statistics, a fast combinatorial algorithm selects a hybrid mix of data‑parallel (DP), tensor‑parallel (TP), and pipeline‑parallel (PP) degrees that best match the workload while respecting device memory constraints.
3. Non‑power‑of‑two handling – The optimizer can split a cluster of, say, 14 NPUs into 3 TP groups of size 4, 4, and 6, avoiding the “round‑up to 16” waste typical of static schemes.
4. Re‑configuration with minimal overhead – The chosen parallelism plan is applied via NCCL/Collective‑Ops primitives; because the optimizer runs in O(N³) time with N = #devices, the whole process adds only a few milliseconds per batch.
5. Integration layer – DHP sits on top of existing training loops, exposing a drop‑in API that automatically swaps the communication topology without requiring model‑level changes.

Results & Findings

• DHP maintains near‑linear scaling up to 64 NPUs even when batch composition swings dramatically (e.g., mixing 4‑K token text with 1024×1024 images).
• The optimizer’s runtime stays under 5 ms per batch, negligible compared with typical forward/backward passes (≈ 30‑50 ms).
• Memory utilization improves by ≈ 10 % because DHP can allocate smaller TP groups for batches that need less model parallelism.

Practical Implications

• Faster model iteration – Teams can train larger multimodal models (e.g., vision‑language or audio‑text) on existing hardware without waiting for static‑parallelism inefficiencies to dominate.
• Cost savings – Higher throughput translates directly into lower cloud‑compute bills; the millisecond‑level optimizer adds virtually no extra cost.
• Flexibility for heterogeneous data pipelines – Data engineers no longer need to pre‑bucket or pad inputs to fit a static parallelism plan, simplifying data ingestion and reducing storage overhead.
• Easier cluster utilization – Organizations with irregularly sized GPU/NPUs clusters (e.g., 14‑node or 22‑node setups) can now achieve high efficiency without artificially inflating node counts to powers of two.
• Plug‑and‑play upgrade – Because DHP works as a thin wrapper around Megatron‑LM/DeepSpeed, existing codebases can adopt it with minimal refactoring, making it attractive for production ML teams.

Limitations & Future Work

• Hardware‑specific tuning – The current implementation is optimized for NPU clusters; performance on heterogeneous GPU/CPU mixes may require additional calibration.
• Scheduler overhead at extreme scale – While the optimizer is fast for up to a few hundred devices, the authors note that scaling to thousands of accelerators could increase planning latency, suggesting a hierarchical or learned scheduler as a next step.
• Model‑specific constraints – Certain architectures (e.g., those with heavy cross‑modal attention) may limit how aggressively TP/PP degrees can be reduced without affecting convergence; further study is needed.
• Open‑source maturity – The reference code is released, but production‑grade robustness (fault tolerance, integration with popular orchestration tools) remains work in progress.

Overall, DHP offers a compelling, developer‑friendly path to squeeze more performance out of existing hardware when training the next generation of multimodal LLMs.

Authors

• Yifan Niu
• Han Xiao
• Dongyi Liu
• Wei Zhou
• Jia Li

Paper Information

• arXiv ID: 2602.21788v1
• Categories: cs.DC, cs.LG
• Published: February 25, 2026
• PDF: Download PDF