[Paper] BAMBO: Construct Ability and Efficiency LLM Pareto Set via Bayesian Adaptive Multi-objective Block-wise Optimization

Source: arXiv - 2512.09972v1

Overview

The paper introduces BAMBO (Bayesian Adaptive Multi‑objective Block‑wise Optimization), a new framework for automatically building a Pareto set that balances ability (e.g., accuracy, language understanding) and efficiency (e.g., latency, memory) of large language models (LLMs). By tackling the “curse of dimensionality” that plagues fine‑grained model merging, BAMBO delivers a richer set of trade‑off solutions than existing coarse‑grained or layer‑wise methods, making it easier for engineers to pick the right model for a given deployment constraint.

Key Contributions

• Hybrid Optimal Block Partitioning: Reformulates the block‑wise merging problem as a 1‑D clustering task and solves it with dynamic programming, dramatically shrinking the search space while preserving important granularity.
• Bayesian Multi‑objective Evolutionary Loop: Integrates the q‑Expected Hypervolume Improvement (qEHVI) acquisition function to guide an evolutionary search toward the most promising ability‑efficiency trade‑offs.
• Automated Pareto Set Construction: Generates a dense, high‑quality Pareto frontier without manual tuning of block sizes or merging strategies.
• Empirical Superiority: Demonstrates on several LLM benchmarks that BAMBO finds Pareto fronts that dominate those produced by state‑of‑the‑art model‑level and layer‑wise baselines.
• Open‑source Release: Provides a ready‑to‑run implementation (https://github.com/xin8coder/BAMBO) for reproducibility and community extension.

Methodology

1. Block‑wise Model Partitioning – Instead of merging whole models (coarse) or individual layers (fine), BAMBO groups consecutive layers into blocks. The optimal block boundaries are discovered by treating the sequence of layers as points on a line and clustering them to balance two objectives:

   • Intra‑block homogeneity: layers inside a block should behave similarly, making them easy to merge.
   • Inter‑block information distribution: blocks should capture distinct functional regions of the network.
     This is solved with a dynamic‑programming algorithm that runs in polynomial time.

2. Bayesian Multi‑objective Search – Each candidate Pareto point corresponds to a specific set of block‑wise merging weights. BAMBO treats the evaluation of a candidate (its ability score and efficiency metric) as a black‑box function and builds a Gaussian‑process surrogate model over the search space.

3. q‑Expected Hypervolume Improvement (qEHVI) – The acquisition function selects a batch of promising candidates that are expected to enlarge the hypervolume (i.e., improve the Pareto front) the most.

4. Evolutionary Loop – The selected candidates are evaluated, the surrogate model is updated, and the process repeats until a budget (e.g., number of evaluations) is exhausted. The final output is a dense set of models spanning the ability‑efficiency trade‑off curve.

Results & Findings

Key takeaways

• Dimensionality reduction via block partitioning makes the search tractable without sacrificing the ability to fine‑tune trade‑offs.
• qEHVI‑driven evolution converges faster than random or grid search, achieving higher hypervolume with fewer model evaluations.
• The resulting Pareto set is dense and diverse, giving developers many more “sweet spots” to choose from.

Practical Implications

• Deploy‑time Flexibility: Teams can instantly pick a model that fits a specific latency budget (e.g., edge device) or memory limit (e.g., mobile) while still meeting a target quality threshold.
• Cost‑aware Scaling: Cloud providers can generate a catalog of LLM variants that trade off GPU hours for inference speed, enabling more granular pricing tiers.
• Rapid Prototyping: Instead of manually experimenting with pruning, quantization, or distillation, developers can run BAMBO once and obtain a ready‑made menu of options.
• Continuous Integration: BAMBO can be slotted into CI pipelines to refresh the Pareto set whenever a new base model or hardware target is added, keeping the model zoo up‑to‑date.
• Research Acceleration: Researchers studying the ability‑efficiency landscape can use the generated frontier as a benchmark for new compression or architecture‑search techniques.

Limitations & Future Work

• Assumption of Block Homogeneity – The clustering step assumes that consecutive layers can be meaningfully grouped; highly irregular architectures may need custom block definitions.
• Surrogate Model Scalability – Gaussian‑process surrogates become expensive beyond a few hundred evaluations; scaling to thousands of candidates may require sparse GP or neural‑network surrogates.
• Metric Coverage – The paper focuses on a single ability metric (e.g., accuracy) and a single efficiency metric (e.g., latency). Extending to multi‑dimensional efficiency (energy, FLOPs, memory) is left for future work.
• Hardware‑Specific Evaluation – All experiments were run on a fixed GPU configuration; real‑world deployment may involve heterogeneous hardware where latency models differ.

Future directions suggested by the authors include adaptive block sizes that react to hardware profiling, integration with quantization/distillation pipelines, and exploring alternative Bayesian acquisition functions for even faster convergence.

Authors

• Kesheng Chen
• Wenjian Luo
• Zhenqian Zhu
• Yamin Hu
• Yiya Xi

Paper Information

• arXiv ID: 2512.09972v1
• Categories: cs.LG, cs.CL, cs.NE
• Published: December 10, 2025
• PDF: Download PDF