[Paper] Generative Classifiers Avoid Shortcut Solutions

Source: arXiv - 2512.25034v1

Overview

The paper Generative Classifiers Avoid Shortcut Solutions demonstrates that classifiers built on class‑conditional generative models (e.g., diffusion or autoregressive models) are far less prone to “shortcut” learning—relying on spurious correlations that break under even modest distribution shift. By modeling the full data distribution rather than just the decision boundary, these generative classifiers achieve state‑of‑the‑art robustness on several image and text benchmarks, opening a practical path for more reliable AI systems in high‑stakes domains.

Key Contributions

• Showcase of generative classifiers as a simple, plug‑and‑play alternative to discriminative models for robustness against spurious features.
• Empirical superiority on five widely used distribution‑shift benchmarks (both vision and language), outperforming strong discriminative baselines without extra augmentations or hyper‑parameter tuning.
• Broad applicability demonstrated on realistic, high‑impact datasets such as medical imaging and satellite imagery, where spurious correlations are common.
• Theoretical insight via a Gaussian toy model that clarifies when and why generative classifiers beat discriminative ones, linking inductive bias to data geometry.
• Practical recipe for turning any pretrained class‑conditional generative model (diffusion, autoregressive, VAE, etc.) into a classifier with minimal overhead.

Methodology

1. Class‑conditional generative modeling – For each class (c), train a generative model (p_\theta(x|c)) that learns to reconstruct the entire input distribution (both core and spurious features).
2. Bayesian inference for classification – At test time, compute posterior probabilities using Bayes’ rule:
   [ \hat{y} = \arg\max_c ; p_\theta(x|c),p(c) ]
   where (p(c)) is a uniform prior or a class‑frequency prior.
3. Model families – The authors experiment with two families:
   • Diffusion models (score‑based generative models) that iteratively denoise a latent noise sample conditioned on the class label.
   • Autoregressive transformers (e.g., GPT‑style) that generate the input token‑by‑token given a class token.
4. Training pipeline – No special regularizers, adversarial augmentations, or knowledge of the spurious correlations are required. The same data used for standard discriminative training is fed to the generative objective (e.g., denoising score matching or next‑token prediction).
5. Evaluation – Robustness is measured on standard distribution‑shift suites (ImageNet‑A/C, Waterbirds, WILDS, etc.) and on domain‑specific tasks (chest X‑ray classification, satellite land‑cover mapping).

Results & Findings

• Robustness boost: Across all tasks, generative classifiers consistently reduce error under distribution shift by 5–10 % absolute.
• Spurious correlation mitigation: In controlled experiments where a nuisance feature is strongly correlated with the label, generative models ignore the nuisance and focus on core semantics, whereas discriminative models overfit to the shortcut.
• Efficiency: Inference cost is roughly 1.5× that of a standard discriminative forward pass (due to likelihood evaluation), but still feasible for batch processing on modern GPUs.

Practical Implications

• Plug‑and‑play robustness: Teams can take an existing class‑conditional diffusion or autoregressive model (many are publicly available) and turn it into a classifier without redesigning the training pipeline.
• Reduced engineering overhead: No need for elaborate data augmentations, adversarial training, or explicit bias‑mitigation heuristics—saving time and compute.
• Higher reliability in regulated domains: Medical diagnostics, remote sensing, and autonomous systems can benefit from classifiers that are less likely to fail when the data distribution drifts (e.g., new scanner models, seasonal satellite imagery).
• Improved interpretability: Because the generative model reconstructs the entire input, developers can inspect generated samples conditioned on each class to understand what features the model deems “core.”
• Potential for hybrid systems: One could combine a fast discriminative front‑end for coarse filtering with a generative classifier for edge cases where robustness is critical.

Limitations & Future Work

• Computational cost: Likelihood evaluation for diffusion models remains heavier than a single discriminative forward pass, which may be prohibitive for latency‑sensitive applications.
• Scalability to very large label spaces: Training a separate generative model per class can become expensive when the number of classes grows into the thousands.
• Dependence on generative quality: If the underlying generative model fails to capture certain fine‑grained details, classification performance may degrade.
• Future directions suggested by the authors include:
  • Developing more efficient likelihood estimators (e.g., amortized scoring) to close the speed gap.
  • Exploring shared‑parameter generative backbones that handle many classes jointly.
  • Extending the analysis to multimodal data (e.g., video + text) and to continual‑learning scenarios where new classes appear over time.

Bottom line: By leveraging the full expressive power of modern generative models, this work offers a surprisingly simple yet powerful recipe for building classifiers that are resilient to the shortcuts that plague conventional discriminative systems. For developers looking to harden AI products against distribution shift, generative classifiers are now a practical tool worth trying.

Authors

• Alexander C. Li
• Ananya Kumar
• Deepak Pathak

Paper Information

• arXiv ID: 2512.25034v1
• Categories: cs.LG, cs.AI, cs.CV, cs.NE
• Published: December 31, 2025
• PDF: Download PDF