[Paper] Measuring the Effect of Background on Classification and Feature Importance in Deep Learning for AV Perception

Source: arXiv - 2512.05937v1

Overview

The paper investigates how background information influences deep‑learning models that recognize traffic signs—a core perception task for autonomous vehicles (AVs). By creating a suite of synthetic sign‑recognition datasets with controlled background‑correlation and camera variations, the authors quantify when and how much a model leans on background cues instead of the sign itself.

Key Contributions

• Systematic synthetic benchmark: Six traffic‑sign datasets that vary only in background‑sign correlation and camera pose, enabling clean isolation of background effects.
• Quantitative metric for background reliance: Extends saliency tools (Grad‑CAM, SHAP) with ground‑truth masks to compute a Background Importance Score (BIS).
• Empirical analysis across model families: Evaluates ResNet‑50, EfficientNet‑B0, and a lightweight MobileNet‑V2 on all datasets, revealing consistent patterns of background dependence.
• Guidelines for dataset design: Shows how camera diversity and background randomization mitigate spurious background learning, offering practical data‑collection recommendations for AV perception pipelines.
• Open‑source release: All synthetic datasets, training scripts, and evaluation code are publicly available at synset.de/datasets/synset-signset-ger/background-effect.

Methodology

1. Synthetic Data Generation – Using a graphics pipeline (Blender + procedural textures), the authors render traffic signs over a set of 30 background scenes. Six variants are produced:
   • Low/High background‑sign correlation (signs placed on a few vs. many backgrounds).
   • Low/High camera variation (fixed front‑on view vs. random yaw/pitch/roll and focal length).
   • Shape‑only control where only sign geometry changes.
2. Model Training – Standard image‑classification pipelines (cross‑entropy loss, Adam optimizer, 100 epochs) are run on each dataset, keeping hyper‑parameters constant across experiments.
3. Explainability Evaluation – For every test image, Grad‑CAM heatmaps and SHAP values are computed. By intersecting these maps with the binary sign mask, the authors derive two numbers:
   • Object Importance (fraction of saliency on the sign).
   • Background Importance Score (BIS = 1 – Object Importance).
4. Statistical Analysis – BIS is aggregated per dataset and model, and correlated with classification accuracy to assess whether higher background reliance hurts or helps performance under different training conditions.

Results & Findings

Key takeaways

• Background correlation boosts raw accuracy when camera viewpoints are limited (the model learns to use the background as a shortcut).
• Increasing camera variation dramatically reduces BIS, forcing the network to attend to the sign itself and slightly lowering accuracy on highly correlated data.
• EfficientNet and MobileNet show the same trend, indicating that the phenomenon is architecture‑agnostic.
• When training and test domains match, background reliance can be harmless; however, under domain shift (e.g., new streets), high BIS leads to a >10 % drop in performance.

Practical Implications

• Dataset design for AV perception – When collecting real‑world sign images, deliberately vary camera angles, lighting, and background scenes to discourage spurious background learning.
• Model validation – Incorporate a background‑importance audit (Grad‑CAM + mask overlap) into CI pipelines; a rising BIS can flag overfitting before deployment.
• Transfer learning strategies – Fine‑tune a model trained on a low‑correlation, high‑variation synthetic set before exposing it to real traffic data; this yields more robust feature representations.
• Edge‑device considerations – Lightweight models (MobileNet‑V2) are equally prone to background shortcuts, so developers cannot rely on model size to avoid the issue.
• Regulatory compliance – Explainability reports that include BIS can satisfy emerging safety standards that demand evidence a vehicle’s perception system bases decisions on relevant objects, not scenery.

Limitations & Future Work

• Synthetic realism – Although the graphics pipeline adds texture variation, the backgrounds still lack the full complexity of real urban scenes (e.g., dynamic occlusions, weather).
• Single‑class focus – The study concentrates on traffic‑sign classification; extending the analysis to multi‑class object detection (pedestrians, vehicles) is needed.
• Static evaluation – Temporal cues (video streams) were not considered; future work could explore how motion information mitigates background reliance.
• Broader XAI tools – Only Grad‑CAM and SHAP were examined; assessing other saliency methods (e.g., LRP, Integrated Gradients) may reveal different sensitivity patterns.

By exposing the hidden role of background pixels in AV perception models, this work equips developers with concrete metrics and data‑collection tactics to build safer, more generalizable autonomous systems.

Authors

• Anne Sielemann
• Valentin Barner
• Stefan Wolf
• Masoud Roschani
• Jens Ziehn
• Juergen Beyerer

Paper Information

• arXiv ID: 2512.05937v1
• Categories: cs.CV, cs.AI, cs.RO
• Published: December 5, 2025
• PDF: Download PDF