[Paper] BAMBO: Construct Ability and Efficiency LLM Pareto Set via Bayesian Adaptive Multi-objective Block-wise Optimization

Source: arXiv - 2512.09972v2

Overview

The paper introduces BAMBO (Bayesian Adaptive Multi‑objective Block‑wise Optimization), a new framework for automatically building a Pareto set of Large Language Models (LLMs) that balances ability (e.g., accuracy, fluency) against efficiency (e.g., latency, memory). By tackling the “curse of dimensionality” that plagues fine‑grained model merging, BAMBO delivers a richer collection of trade‑off models that developers can pick from to meet diverse deployment constraints.

Key Contributions

• Hybrid Optimal Block Partitioning: Reformulates the layer‑wise merging problem as a 1‑D clustering task and solves it with dynamic programming, dramatically cutting the search space while preserving important granularity.
• Bayesian Multi‑objective Evolutionary Loop: Integrates the q‑Expected Hypervolume Improvement (qEHVI) acquisition function to guide the search toward high‑quality ability‑efficiency trade‑offs.
• Automated Pareto Set Construction: Generates a comprehensive set of merged LLMs without manual tuning, enabling rapid model selection for different hardware or latency budgets.
• Empirical Superiority: Shows that BAMBO discovers a more extensive and higher‑quality Pareto frontier compared with existing coarse‑grained (model‑level) and fine‑grained (layer‑wise) baselines.
• Open‑source Release: Provides a ready‑to‑use implementation (https://github.com/xin8coder/BAMBO) for the community.

Methodology

1. Block‑wise Partitioning – Instead of merging whole models or individual layers, BAMBO groups consecutive layers into blocks. The optimal block boundaries are found by treating the problem as a 1‑D clustering task: a dynamic‑programming algorithm evaluates candidate partitions, balancing intra‑block similarity (so blocks are homogeneous) against inter‑block information spread (so each block still carries distinct knowledge).
2. Search Space Reduction – By merging at the block level, the dimensionality drops from thousands of layer‑wise decisions to a handful of block‑wise decisions, making the optimization tractable.
3. Bayesian Multi‑objective Optimization – An evolutionary loop proposes new block‑wise merge configurations. Each candidate is evaluated on two objectives: (a) ability (e.g., perplexity, downstream task accuracy) and (b) efficiency (e.g., FLOPs, inference latency). The qEHVI acquisition function predicts which candidates will most improve the hypervolume of the current Pareto front, steering the search toward promising regions.
4. Iterative Refinement – The loop repeats: evaluate, update the surrogate model, select new candidates, and expand the Pareto set until convergence or a budget limit.

Results & Findings

• Broader Frontier: BAMBO’s Pareto front contains 30‑40 % more non‑dominated models than the best baseline, covering a wider span of latency‑accuracy trade‑offs.
• Higher Quality Points: On benchmark tasks (e.g., GLUE, WikiText), the best BAMBO models achieve up to 1.2 % lower perplexity while using 15 % fewer FLOPs compared with the strongest layer‑wise merging baseline.
• Search Efficiency: Thanks to block partitioning, the total number of evaluated configurations drops by an order of magnitude, reducing GPU hours required for Pareto construction.
• Robustness: The method works across different model families (e.g., GPT‑2, LLaMA) and scales to models with >10 B parameters.

Practical Implications

• Tailored Deployments: Teams can instantly pick a model from the BAMBO‑generated Pareto set that matches their hardware budget—e.g., a low‑latency edge device vs. a high‑throughput cloud service.
• Cost‑Effective Fine‑tuning: Instead of training multiple variants from scratch, developers can merge existing checkpoints to meet new constraints, saving compute and time.
• Rapid Prototyping: The open‑source tool integrates with popular libraries (Hugging Face Transformers), allowing engineers to plug in their own models and constraints with minimal code changes.
• Product Road‑mapping: Product managers can visualize ability‑efficiency trade‑offs quantitatively, making informed decisions about which model size to ship for a given SLA.

Limitations & Future Work

• Block Granularity Trade‑off: While block‑wise merging reduces dimensionality, it may miss some ultra‑fine‑grained interactions that only layer‑level merging can capture.
• Evaluation Cost: Accurate ability metrics still require running inference on validation data, which can be expensive for very large models.
• Scope of Objectives: The current formulation focuses on ability and FLOP‑based efficiency; extending to other metrics (e.g., memory footprint, energy consumption) is left for future research.
• Generalization to Non‑Transformer Architectures: The method is demonstrated on transformer‑based LLMs; applying it to other architectures (e.g., retrieval‑augmented models) remains an open question.

BAMBO opens a practical pathway for developers to navigate the ever‑tightening ability‑efficiency curve of LLMs, turning what was once a manual, trial‑and‑error process into an automated, data‑driven workflow.

Authors

• Kesheng Chen
• Wenjian Luo
• Zhenqian Zhu
• Yamin Hu
• Yiya Xi

Paper Information

• arXiv ID: 2512.09972v2
• Categories: cs.LG, cs.CL, cs.NE
• Published: December 10, 2025
• PDF: Download PDF