[Paper] BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations

Source: arXiv - 2603.06576v1

Overview

The paper BEVLM introduces a novel way to fuse the powerful reasoning of Large Language Models (LLMs) with the spatial precision of Bird’s‑Eye View (BEV) maps used in autonomous driving. By distilling semantic knowledge from LLMs directly into BEV representations, the authors achieve far more consistent cross‑view reasoning and a noticeable boost in driving safety metrics.

Key Contributions

• Unified BEV‑LLM Interface: A single BEV feature tensor is fed to the LLM, eliminating the need for separate per‑camera token streams and preserving geometric coherence.
• Semantic Distillation Pipeline: A two‑stage training scheme that first teaches the LLM to interpret BEV inputs, then transfers its semantic understanding back into the BEV encoder.
• Large‑Scale Empirical Gains: Demonstrated a 46 % improvement in LLM‑driven scene reasoning accuracy and a 29 % lift in closed‑loop end‑to‑end driving safety on challenging benchmarks.
• Cross‑Domain Generalization: Shows that the distilled BEV representations retain semantic richness even when deployed on unseen towns, weather, and traffic conditions.

Methodology

1. BEV Feature Extraction – A conventional perception stack (camera → multi‑view transformer → BEV encoder) produces a dense top‑down map containing object locations, lane geometry, and depth cues.
2. LLM Conditioning – The BEV map is flattened into a sequence of visual tokens and concatenated with a textual prompt (e.g., “What is the safest lane to change into?”). The LLM processes this joint sequence using its standard transformer layers.
3. Bidirectional Distillation
   • LLM‑to‑BEV: The LLM’s hidden states are projected back onto the BEV grid, teaching the BEV encoder to embed high‑level semantics (intent, affordances).
   • BEV‑to‑LLM: Simultaneously, the BEV encoder is fine‑tuned to produce token embeddings that the LLM can interpret without extra adapters.
4. Training Regime – The authors use a mixture of supervised driving logs (for geometry) and language‑grounded tasks (e.g., question answering, instruction following) to jointly optimize both modules.

Results & Findings

• Spatial Consistency: Because the LLM sees a single, globally aligned BEV map, its answers remain coherent across frames and viewpoints.
• Semantic Enrichment: The distilled BEV features encode not just “car at (x,y)” but also “car is likely to turn left,” enabling richer planning.
• Robustness: Tests on simulated night‑time and heavy‑rain scenarios show only a modest drop (<5 %) compared with a 20 % drop for the baseline.

Practical Implications

• Simplified Perception Pipelines: Engineers can replace multiple camera‑specific tokenizers with a single BEV encoder, reducing code complexity and GPU memory usage.
• Better Human‑Vehicle Interaction: The unified BEV‑LLM interface makes it straightforward to add natural‑language commands (“take the next exit”) without redesigning the perception stack.
• Safety‑Critical Decision Making: The semantic enrichment of BEV maps can be directly consumed by downstream planners or motion‑prediction modules, leading to more cautious and explainable maneuvers.
• Transferability: Since BEVLM learns language‑level concepts (e.g., “school zone,” “construction”), the same model can be fine‑tuned for new jurisdictions with minimal additional data.

Limitations & Future Work

• Dependence on High‑Quality BEV Ground Truth: The distillation process assumes accurate geometric annotations; noisy sensor setups may degrade performance.
• Scalability to Full‑Stack Autonomy: The paper focuses on perception‑to‑LLM reasoning; integrating with long‑horizon planning and control loops remains an open challenge.
• Real‑World Validation: Experiments are conducted in high‑fidelity simulators; field trials on real vehicles are needed to confirm latency and robustness claims.
• Future Directions: The authors suggest extending BEVLM to multimodal LLMs (audio, map data), exploring self‑supervised BEV pre‑training, and evaluating on diverse hardware (edge GPUs, automotive ASICs).

Authors

• Thomas Monninger
• Shaoyuan Xie
• Qi Alfred Chen
• Sihao Ding

Paper Information

• arXiv ID: 2603.06576v1
• Categories: cs.CV, cs.AI, cs.LG, cs.RO
• Published: March 6, 2026
• PDF: Download PDF