[Paper] Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Driving

Source: arXiv - 2602.09018v1

Overview

This paper tackles a core problem for autonomous‑driving AI: how well a vision‑based driving policy survives when the world looks different from the data it was trained on (out‑of‑distribution, OOD). Rather than collapsing robustness to a single accuracy number, the authors systematically vary five environmental factors—scene type, season, weather, time of day, and the mix of traffic agents—and measure how each factor (and combinations of them) affect closed‑loop driving performance in the VISTA simulator.

Key Contributions

• Factorized OOD benchmark – Introduces a controlled “k‑factor” perturbation framework (k = 0…3) that isolates the impact of individual and combined environment changes.
• Comprehensive model comparison – Evaluates fully‑connected (FC), convolutional (CNN), and Vision‑Transformer (ViT) policies, including lightweight ViT heads built on frozen foundation‑model (FM) features.
• Empirical robustness hierarchy – Shows ViT‑based policies consistently outperform comparable CNN/FC models on OOD scenarios, and FM‑feature policies achieve state‑of‑the‑art success rates (with a modest latency trade‑off).
• Quantified factor impact – Identifies the biggest single‑factor drops: rural → urban and day → night (~31 % each), followed by actor swaps (~10 %) and moderate rain (~7 %).
• Non‑additive factor interactions – Demonstrates that some factor pairings mitigate each other while others (e.g., season + time) compound the degradation.
• Training‑data design rules – Finds that exposing the model to winter/snow conditions yields the most robust single‑factor performance, while a mixed rural‑summer baseline gives the best overall OOD resilience.
• Scaling vs. targeted exposure – Shows that increasing the number of training traces (5 → 14) improves robustness (+11.8 % success), but carefully curated hard‑condition samples can achieve similar gains with less data.
• Multi‑ID training benefits – Training on several in‑distribution (ID) environments broadens coverage (urban OOD success ↑ from 60.6 % to 70.1 %) with only a small drop on ID performance.

Methodology

1. Environment factorization – The authors define five orthogonal axes:

   • Scene: rural vs. urban road layouts
   • Season: summer vs. winter (snow)
   • Weather: clear vs. moderate rain
   • Time: day vs. night
   • Agent mix: different traffic participant densities/types

2. k‑factor perturbations – For each test, they flip 0, 1, 2, or 3 of the axes simultaneously, creating a controlled OOD difficulty ladder.

3. Simulation platform – All experiments run in the VISTA closed‑loop driving simulator, which executes the policy’s steering/throttle commands and measures success (completion of a predefined route without infractions).

4. Model families –

   • FC: shallow fully‑connected networks on raw image pixels.
   • CNN: classic convolutional backbones (e.g., ResNet‑18).
   • ViT: Vision Transformers of comparable parameter count.
   • FM‑feature ViT: A frozen large‑scale foundation model (e.g., CLIP‑ViT) provides image embeddings; a tiny trainable head (few layers) maps embeddings to driving actions.

5. Training variations – They manipulate three dimensions of the ID training set:

   • Scale: number of driving traces (5 → 14).
   • Diversity: inclusion of multiple scenes, seasons, etc.
   • Temporal context: single‑frame vs. multi‑frame inputs (the latter proved ineffective).

6. Metrics – Primary metric is success rate (percentage of routes completed without collision or rule violation). Secondary metrics include inference latency.

Results & Findings

• ViT vs. CNN/FC: ViT policies gain ~8–12 % higher success under the toughest 3‑factor OOD tests.
• FM‑feature heads: Achieve the highest absolute OOD success (≈ 90 % on 3‑factor tests) but incur ~2 ms extra latency per inference.
• Temporal inputs: Adding previous frames does not improve over the best single‑frame ViT baseline, suggesting that current architectures already capture sufficient spatial cues.
• Training on winter/snow: Provides the strongest single‑factor robustness (especially against season changes).
• Rural + summer baseline: Yields the best average OOD performance across all factor combinations.
• Scaling traces: Moving from 5 to 14 traces lifts average OOD success by ~11.8 percentage points.
• Multi‑ID training: Improves performance on OOD urban scenarios by ~9.5 % with only a ~2 % drop on ID performance.

Practical Implications

• Model selection: For production autonomous‑driving stacks, Vision Transformers (especially when paired with frozen foundation‑model embeddings) are a pragmatic choice for OOD resilience, even if they cost a few milliseconds more per frame.
• Data collection strategy: Rather than amassing massive amounts of homogeneous driving data, teams should prioritize diverse conditions—especially winter/snow and a mix of rural/urban scenes—to obtain the biggest robustness gains per annotation hour.
• Testing pipelines: The k‑factor perturbation framework can be integrated into CI for autonomous‑driving software, automatically surfacing which environmental changes cause the biggest performance cliffs.
• Latency budgeting: The modest latency increase of FM‑feature policies can be mitigated by hardware acceleration (e.g., TensorRT, ONNX Runtime) or by using a lightweight head that runs on a separate edge processor.
• Temporal modeling: Since naïve multi‑frame inputs didn’t help, developers should invest in more sophisticated temporal architectures (e.g., attention over a learned motion representation) if they need to capture dynamics beyond what a single frame provides.
• Robustness‑by‑design: The non‑additive nature of factor interactions suggests that robustness testing must consider combinations of conditions, not just isolated ones—critical for safety certification.

Limitations & Future Work

• Simulator fidelity: All experiments are confined to the VISTA simulator; real‑world transfer may reveal additional failure modes.
• Latency trade‑off: The paper reports latency but does not explore aggressive model compression or quantization that could close the gap for FM‑feature policies.
• Temporal modeling: Only simple multi‑frame concatenation was tested; more advanced recurrent or transformer‑based temporal encoders remain unexplored.
• Factor granularity: The five axes are coarse (e.g., “moderate rain” vs. heavy rain); finer granularity could uncover subtler robustness patterns.
• Safety metrics: Success rate is a high‑level metric; future work could incorporate more nuanced safety indicators (time‑to‑collision, lateral deviation, etc.).

Bottom line: By breaking down OOD robustness into interpretable factors and rigorously benchmarking modern vision models, the study offers concrete, data‑driven guidance for building more resilient autonomous‑driving perception and control pipelines.

Authors

• Amir Mallak
• Alaa Maalouf

Paper Information

• arXiv ID: 2602.09018v1
• Categories: cs.RO, cs.AI, cs.CV, cs.LG
• Published: February 9, 2026
• PDF: Download PDF