[Paper] Rethinking the Spatio-Temporal Alignment of End-to-End 3D Perception

Source: arXiv - 2512.23635v1

Overview

The paper introduces HAT, a novel spatio‑temporal alignment module that lets each detected object pick the best motion hypothesis from a set of explicit motion models. By combining motion‑aware proposals with semantic cues, HAT dramatically improves 3D perception and tracking for autonomous‑driving pipelines, especially when visual cues are noisy or corrupted.

Key Contributions

• Multi‑hypothesis alignment: Generates several motion‑based spatial anchors (e.g., constant‑velocity, constant‑acceleration) for each historic object and lets the network select the most suitable one without direct supervision.
• Motion‑aware feature proposals: Couples each anchor with a feature vector that encodes both appearance and motion information, enabling richer temporal reasoning.
• Plug‑and‑play design: HAT can be inserted into any end‑to‑end 3D detector or tracker (DETR3D, BEVFormer, etc.) and yields consistent gains.
• State‑of‑the‑art tracking: Achieves 46.0 % AMOTA on the nuScenes test split, surpassing previous methods.
• Robustness to corrupted semantics: Demonstrates that stronger motion modeling reduces perception errors and downstream planning collisions by up to 32 % in the nuScenes‑C benchmark.

Methodology

1. Historical query cache: For each object detected in previous frames, the system stores a query containing its semantic embedding and a rough motion estimate.
2. Explicit motion models: A small library of deterministic motion hypotheses (e.g., constant velocity, constant turn rate) projects the cached query forward to the current frame, producing multiple spatial anchors.
3. Feature proposal generation: Each anchor is paired with a motion‑aware feature vector that fuses the original semantic embedding with the hypothesized motion.
4. Multi‑hypothesis decoding: A lightweight attention decoder consumes the set of proposals and the current frame’s queries, scoring each hypothesis with learned compatibility weights. The highest‑scoring proposal becomes the final alignment for that object.
5. End‑to‑end training: The whole pipeline is trained with the standard detection/tracking losses; the hypothesis selection emerges implicitly because the loss penalizes mis‑aligned predictions.

The approach sidesteps the need for a single, hand‑crafted motion model and lets the network learn when a more complex or a simpler motion description is appropriate.

Results & Findings

Across several detector backbones, HAT consistently lifts performance, confirming that explicit motion hypotheses complement semantic attention mechanisms. The biggest gains appear when semantic cues are degraded, highlighting the module’s ability to fall back on motion consistency.

Practical Implications

• Plug‑in upgrade for existing stacks: Autonomous‑driving perception pipelines that already use transformer‑based detectors can adopt HAT with minimal code changes, gaining immediate tracking accuracy and safety benefits.
• Better planning under sensor degradation: In adverse weather or sensor failure scenarios, the motion‑driven alignment keeps object trajectories stable, reducing false positives/negatives that would otherwise trigger unsafe maneuvers.
• Reduced reliance on heavy LiDAR/Camera fusion: Because HAT extracts more value from temporal consistency, developers can achieve comparable performance with sparser sensor setups, potentially lowering hardware costs.
• Scalable to edge devices: The hypothesis decoder is lightweight (a few attention heads), making it feasible for real‑time inference on automotive‑grade GPUs or specialized accelerators.
• Foundation for predictive modules: The explicit motion hypotheses can be extended to forecast future states, feeding downstream prediction and decision‑making modules with higher‑quality inputs.

Limitations & Future Work

• Hypothesis library size: The current set of motion models is handcrafted and limited; adding more complex dynamics (e.g., slip, variable acceleration) could further improve rare‑case handling but may increase computational load.
• Dependence on accurate historical queries: If the cache contains badly mis‑localized objects, the generated anchors may mislead the decoder; robust cache management strategies are needed.
• Evaluation limited to nuScenes: While results are strong on this benchmark, broader validation on other datasets (Waymo Open, Argoverse) and real‑world fleets would solidify generalizability.
• Integration with sensor‑fusion pipelines: Future work could explore joint optimization of HAT with radar or map‑based priors, enabling richer context‑aware motion modeling.

Overall, HAT offers a practical, performance‑boosting upgrade for end‑to‑end 3D perception systems, bridging the gap between classic motion modeling and modern attention‑driven architectures.

Authors

• Xiaoyu Li
• Peidong Li
• Xian Wu
• Long Shi
• Dedong Liu
• Yitao Wu
• Jiajia Fu
• Dixiao Cui
• Lijun Zhao
• Lining Sun

Paper Information

• arXiv ID: 2512.23635v1
• Categories: cs.CV
• Published: December 29, 2025
• PDF: Download PDF