[Paper] On the Bayes Inconsistency of Disagreement Discrepancy Surrogates

Source: arXiv - 2512.05931v1

Overview

The paper investigates a subtle but critical flaw in how researchers currently train models to detect distribution shift using the disagreement discrepancy metric. While the metric itself is powerful for spotting when two models start to disagree under new data, the surrogate loss functions used to optimize it are shown to be Bayes inconsistent—meaning they can’t reliably maximize the true metric. The authors not only prove this inconsistency but also propose a new, provably consistent surrogate that works hand‑in‑hand with standard cross‑entropy loss.

Key Contributions

• Theoretical proof of Bayes inconsistency for all existing surrogate losses used to optimize disagreement discrepancy.
• Upper‑ and lower‑bound analysis of the optimality gap, quantifying how far off current surrogates can be from the true objective.
• Design of a novel “disagreement loss” that, when combined with cross‑entropy, is provably Bayes consistent for the disagreement discrepancy metric.
• Extensive empirical validation on several benchmark datasets (including adversarially perturbed and naturally shifted corpora) showing tighter estimates of shift and more robust model performance.
• Practical guidelines for integrating the new loss into existing training pipelines without heavy computational overhead.

Methodology

1. Formalizing Disagreement Discrepancy – The authors define the metric as the expected change in the zero‑one disagreement between two classifiers when moving from a source distribution (P) to a target distribution (Q).
2. Surrogate Loss Landscape – Because the zero‑one loss is non‑differentiable, prior work replaces it with smooth surrogates (e.g., hinge, logistic). The paper treats each surrogate as a risk and studies its Bayes optimal classifier.
3. Inconsistency Proof – By constructing counter‑examples, the authors demonstrate that the Bayes optimal classifier for any of the existing surrogates does not coincide with the classifier that maximizes the true disagreement discrepancy.
4. Bounding the Gap – Using tools from statistical learning theory, they derive tight upper and lower bounds on the difference between surrogate risk and the true risk, giving a quantitative measure of the inconsistency.
5. New Consistent Surrogate – They introduce a loss that directly penalizes pairwise disagreement predictions while preserving differentiability. When added to the standard cross‑entropy objective, the combined loss satisfies the Bayes consistency condition.
6. Empirical Pipeline – Experiments follow a standard “train‑two‑models‑and‑measure‑disagreement” workflow, swapping in the new loss for the old surrogate and evaluating on shift detection, error bounding, and robustness tasks.

Results & Findings

• The new loss consistently yields tighter estimates of the true disagreement discrepancy, especially under aggressive adversarial perturbations where prior surrogates collapse.
• Models trained with the combined loss show higher robustness to unseen shifts, reflected in lower downstream classification error on the shifted data.
• The computational overhead is negligible (< 2 % extra runtime) because the loss can be computed from the same logits used for cross‑entropy.

Practical Implications

• Shift Detection Services – Cloud providers can replace their current surrogate‑based monitors with the new loss, gaining more reliable alerts before model performance degrades.
• Robust Model Training – Adding the disagreement loss to existing training scripts is as simple as appending an extra term to the loss function; no architectural changes are required.
• Safety‑Critical Systems – In domains like autonomous driving or medical imaging, tighter disagreement bounds translate to better guarantees that a model will not silently fail under distribution drift.
• Tooling & Libraries – The authors release a lightweight PyTorch implementation; integrating it into popular libraries (e.g., torchvision, scikit‑learn) would let developers experiment with minimal friction.

Limitations & Future Work

• The consistency proof assumes i.i.d. samples from source and target distributions; real‑world streaming data may violate this.
• Experiments focus on image classification; extending to NLP, time‑series, or reinforcement‑learning settings remains open.
• The current formulation requires two separate models; future work could explore single‑model approximations to reduce memory usage.
• The authors note that tighter bounds might be achievable by jointly learning the shift estimator and the classifiers, a direction they plan to investigate.

Authors

• Neil G. Marchant
• Andrew C. Cullen
• Feng Liu
• Sarah M. Erfani

Paper Information

• arXiv ID: 2512.05931v1
• Categories: cs.LG, stat.ML
• Published: December 5, 2025
• PDF: Download PDF