[Paper] Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models

Source: arXiv - 2604.24708v1

Overview

Training massive neural networks usually means running many identical GPU replicas in lock‑step, all following the same learning‑rate schedule. The new Hyperparameter‑Divergent Ensemble Training (HDET) framework flips that paradigm: it lets each replica explore a different learning‑rate (or other scalar hyperparameter) while still sharing the same model weights. By periodically averaging the weights, HDET discovers a high‑performing schedule on‑the‑fly without extra compute or costly hyperparameter sweeps.

Key Contributions

• Ensemble‑based learning‑rate exploration: Repurposes data‑parallel GPUs to run divergent learning‑rate schedules in parallel, incurring only the cheap AllReduce communication used for weight averaging.
• Fan‑out / converge protocol: Alternates between independent “fan‑out” phases (each replica follows a symmetric spread of rates) and synchronized “converge” phases (weights are averaged every (T) steps).
• Automatic LR controller (auto‑LR): Uses the relative loss across replicas as a zero‑order performance signal and updates a shared base schedule with a momentum‑based meta‑update, eliminating manual LR tuning.
• General‑purpose scalar hyperparameter search: The same mechanism works for dropout, weight‑decay, temperature scaling, etc., treating loss differences as hypergradients.
• Drop‑in PyTorch implementation: Provided as a replacement for OneCycleLR, requiring no changes to model code, optimizer, or data pipeline.

Methodology

1. Initialization – All (N) replicas start from the same model parameters and a common “base” learning‑rate schedule.
2. Fan‑out stage – The base schedule is symmetrically perturbed for each replica (e.g., ( \eta_i = \eta_{\text{base}} \times (1 + \delta_i) ) with (\delta_i) spread evenly around zero). Replicas train independently for (T_{\text{fan}}) steps, each logging its training loss.
3. Converge stage – After the fan‑out window, an AllReduce operation averages the model weights across all replicas, synchronizing them back to a common state.
4. Auto‑LR meta‑update – The relative losses (\ell_i) are turned into a gradient‑free signal: replicas with lower loss indicate a beneficial direction for the base schedule. A momentum update adjusts the base schedule toward the “winning” perturbations.
5. Repeat – The process cycles between fan‑out and converge until training finishes.

Because the only extra communication is the weight averaging already required for data‑parallel SGD, the overhead is negligible. The algorithm can be visualized as a “ring of explorers” that periodically meet to share their discoveries.

Results & Findings

Key observations

• Optimization quality improves: The auto‑LR schedule converges faster (≈ 10 % fewer epochs to reach the same loss) because the controller quickly homes in on a near‑optimal LR curve.
• Generalization boost: Slightly higher validation accuracy / lower perplexity suggests that the stochastic LR diversity acts as a regularizer.
• Negligible extra cost: Wall‑clock time increased by < 2 % compared with vanilla data‑parallel training, confirming the low communication overhead.

Practical Implications

• Eliminate manual LR sweeps – Teams can launch a single training run and let HDET discover a competitive schedule, saving weeks of experimentation on large clusters.
• Leverage idle parallelism – In environments where GPUs are already allocated for data parallelism (e.g., multi‑node training), HDET turns those replicas into a built‑in hyperparameter search engine.
• Plug‑and‑play for any scalar hyperparameter – Drop‑in replacement for OneCycleLR means you can simultaneously explore dropout rates, weight‑decay, or temperature scaling without writing custom search loops.
• Potential for AutoML pipelines – HDET’s zero‑order meta‑updates fit naturally into automated training pipelines, providing a lightweight alternative to Bayesian optimization or population‑based training for large models.
• Reduced carbon footprint – By avoiding multiple full‑scale training runs, organizations can cut the energy consumption associated with hyperparameter tuning.

Limitations & Future Work

• Scalability to extreme replica counts – The current study uses up to 16 GPUs; very large ensembles may suffer from diminishing returns as the perturbation space becomes crowded.
• Assumption of smooth loss landscape – The momentum‑based meta‑update works best when loss differences across LR perturbations are monotonic; highly noisy or non‑convex regimes could mislead the controller.
• Fixed perturbation pattern – HDET currently uses a symmetric spread; adaptive or learned perturbation distributions could improve exploration efficiency.
• Extension beyond scalar hyperparameters – Future work could investigate joint exploration of multiple hyperparameters (e.g., LR + weight‑decay) or architectural choices that still permit weight averaging.

Overall, HDET offers a pragmatic, low‑overhead path to automatic learning‑rate (and scalar hyperparameter) optimization for today’s large‑scale deep‑learning workloads.

Authors

• Hailing Cheng
• Tao Huang
• Chen Zhu
• Antonio Alonso

Paper Information

• arXiv ID: 2604.24708v1
• Categories: cs.LG, cs.AI
• Published: April 27, 2026
• PDF: Download PDF