[Paper] ACE-Brain-0: Spatial Intelligence as a Shared Scaffold for Universal Embodiments

Source: arXiv - 2603.03198v1

Overview

The paper introduces ACE‑Brain‑0, a multimodal large language model (MLLM) that can reason about space and control a wide range of embodied agents—from self‑driving cars to warehouse robots and UAVs. By treating spatial intelligence as a universal “scaffold,” the authors show how a single model can learn to navigate, manipulate, and plan across dramatically different physical platforms without sacrificing performance on any single domain.

Key Contributions

• Spatial Scaffold Concept: Demonstrates that 3‑D spatial reasoning is a domain‑agnostic foundation that can be shared across heterogeneous embodiments.
• Scaffold‑Specialize‑Reconcile (SSR) Paradigm: A three‑stage training pipeline that first builds a common spatial core, then fine‑tunes domain‑specific experts, and finally merges them without additional data.
• Group Relative Policy Optimization (GRPO): A novel policy‑learning algorithm that balances the competing objectives of multiple embodiments during joint training.
• Unified Evaluation Suite: Benchmarks ACE‑Brain‑0 on 24 tasks spanning autonomous driving, robotic manipulation, UAV navigation, and pure spatial reasoning, achieving state‑of‑the‑art results on several of them.
• Data‑Free Model Merging: Introduces a lightweight, memory‑efficient method to combine specialist models into a single deployable brain, sidestepping catastrophic forgetting.

Methodology

1. Scaffold Phase – The model is first trained on large‑scale 3‑D perception and reasoning datasets (e.g., point‑cloud captioning, map‑to‑text, geometry QA). This builds a spatial intelligence core that learns to encode and query 3‑D environments in a language‑friendly format.

2. Specialize Phase – Separate “expert” heads are attached for each embodiment type (car, robot arm, drone). Using reinforcement learning from human feedback (RLHF) and domain‑specific simulators, each expert learns policies that map the shared spatial representation to actions (steering, joint torques, thrust vectors).

3. Reconcile Phase – Instead of fine‑tuning the whole model again (which would cause interference), the authors employ data‑free model merging: they align the weight distributions of the experts via a lightweight linear interpolation guided by a compatibility matrix derived from the scaffold’s parameters. This yields a single, compact model that retains each expert’s proficiency.

4. GRPO Optimizer – During the specialist training, GRPO treats each embodiment as a “group” and optimizes a relative policy objective that penalizes performance drops on other groups, thereby reducing gradient interference and preserving the shared scaffold.

All components are built on top of a transformer‑based MLLM (similar to LLaMA‑2) with vision‑language adapters, making the system compatible with existing inference stacks.

Results & Findings

• Cross‑embodiment transfer: When fine‑tuned on a single domain, the model retains >85% performance on the other two domains, confirming the scaffold’s robustness.
• Ablation: Removing GRPO drops multi‑domain performance by ~7 points, while skipping the Reconcile step leads to catastrophic forgetting in two of the three domains.
• Inference footprint: The merged model fits in 12 GB VRAM (FP16), enabling real‑time inference on a single RTX 4090 for all three embodiment types.

Practical Implications

• One‑Model Deployment: Companies can ship a single AI package that powers autonomous cars, warehouse robots, and inspection drones, reducing engineering overhead and maintenance costs.
• Rapid Prototyping: Developers can plug in a new embodiment (e.g., a delivery robot) by training only a lightweight specialist head, leveraging the existing spatial scaffold for immediate competence.
• Safety & Consistency: A shared spatial core ensures that safety constraints (e.g., collision avoidance) are uniformly enforced across platforms, simplifying certification pipelines.
• Edge‑Ready: The data‑free merging technique avoids the need for massive multi‑task datasets on device, making it feasible for on‑device updates in robotics fleets.

Limitations & Future Work

• Simulation‑to‑Real Gap: All experiments are conducted in high‑fidelity simulators; real‑world validation (especially for UAVs in windy conditions) remains an open challenge.
• Scaffold Generality: While 3‑D reasoning works well for ground and aerial vehicles, modalities like soft‑robotic manipulation or underwater navigation may require additional sensory scaffolds (e.g., fluid dynamics).
• Scalability of Experts: Adding many more embodiments could increase the parameter budget; future research will explore modular sparsity or mixture‑of‑experts to keep the model lightweight.
• Explainability: The current model treats the spatial scaffold as a black box; integrating explicit geometric reasoning (e.g., symbolic maps) could improve interpretability for safety‑critical deployments.

ACE‑Brain‑0 marks a significant step toward truly universal embodied AI, showing that a well‑designed spatial foundation can serve as the common language between cars, robots, and drones—opening the door to more flexible, maintainable, and scalable intelligent systems.

Authors

• Ziyang Gong
• Zehang Luo
• Anke Tang
• Zhe Liu
• Shi Fu
• Zhi Hou
• Ganlin Yang
• Weiyun Wang
• Xiaofeng Wang
• Jianbo Liu
• Gen Luo
• Haolan Kang
• Shuang Luo
• Yue Zhou
• Yong Luo
• Li Shen
• Xiaosong Jia
• Yao Mu
• Xue Yang
• Chunxiao Liu
• Junchi Yan
• Hengshuang Zhao
• Dacheng Tao
• Xiaogang Wang

Paper Information

• arXiv ID: 2603.03198v1
• Categories: cs.RO, cs.CL, cs.CV
• Published: March 3, 2026
• PDF: Download PDF