[Paper] Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop

Source: arXiv - 2601.05184v1

Overview

Large language models (LLMs) are increasingly used to generate synthetic data that later trains the next generation of models. This creates a self‑consuming performative loop (SCPL): a model’s own outputs become part of its training set, and the loop can amplify hidden biases. The paper by Wang et al. systematically studies how such loops affect bias and proposes a simple, reward‑driven sampling technique to keep the system trustworthy.

Key Contributions

• Formalization of SCPL – Introduces the notion of a self‑consuming performative loop and distinguishes two realistic training regimes: full‑model retraining and incremental fine‑tuning.
• Controlled experimental framework – Builds a sandbox that mimics feedback‑driven data generation while keeping user preference data private, enabling clean measurement of bias evolution.
• Empirical bias analysis – Shows that, across three downstream tasks, the performative loop increases preference bias (the model favors the majority’s preferences) while reducing disparate bias (differences across protected groups).
• Reward‑based rejection sampling – Proposes a lightweight mitigation: during data generation, samples are accepted with probability proportional to a bias‑aware reward, curbing the growth of preference bias.
• Open‑source implementation – Releases code and synthetic datasets to facilitate reproducibility and future research on bias‑aware self‑improving LLM pipelines.

Methodology

1. Loop Simulation

   • Start with a seed LLM (the “base model”).
   • Generate synthetic responses to a set of prompts.
   • Score each response with a reward model that captures user preference (e.g., relevance, helpfulness).
   • Select a subset of responses using rejection sampling: higher‑reward samples are more likely to be kept.
   • Add the selected synthetic pairs to the training corpus and retrain (full retraining) or fine‑tune (incremental) the LLM.
   • Repeat the cycle for several iterations, mimicking a production system that continuously learns from its own output.

2. Bias Measurement

   • Preference bias: disparity in model scores between majority‑aligned and minority‑aligned prompts.
   • Disparate bias: performance gaps across protected attributes (e.g., gender, ethnicity) measured with standard fairness metrics (e.g., equalized odds, demographic parity).

3. Tasks & Datasets

   • Sentiment classification, open‑ended question answering, and code generation—each with annotated demographic sub‑groups to evaluate bias.

4. Mitigation Strategy

   • Define a bias‑aware reward = original reward – λ·bias_penalty, where the penalty reflects how much a sample would exacerbate preference bias.
   • Use this reward in the rejection sampler, effectively down‑weighting “biased” synthetic examples before they re‑enter the training loop.

Results & Findings

• Preference bias grows noticeably after each loop, especially in full retraining where the model fully absorbs its own biased outputs.
• Disparate bias slightly shrinks, likely because the synthetic data becomes more homogeneous across demographic groups.
• The reward‑based rejection sampling dramatically curtails the rise of preference bias while preserving (or even slightly improving) overall task performance.

Practical Implications

• Production pipelines that continuously fine‑tune LLMs on user‑generated content should monitor bias metrics each iteration; otherwise, hidden preference bias can silently accumulate.
• The reward‑based rejection sampler is easy to drop into existing data‑generation workflows (it only requires a bias‑aware scoring function), offering a low‑overhead guardrail.
• Companies building LLM‑as‑a‑service can adopt the incremental fine‑tuning regime combined with bias‑aware sampling to reap the benefits of rapid model updates without sacrificing fairness.
• The findings suggest that synthetic data alone is not a silver bullet; developers need to blend it with curated, human‑annotated examples or apply debiasing post‑hoc to keep the system trustworthy.

Limitations & Future Work

• The study uses synthetic reward models as proxies for real user preferences; actual user feedback may be noisier or exhibit different bias patterns.
• Experiments are limited to three tasks and a handful of demographic attributes; broader domain coverage (e.g., multilingual settings) remains unexplored.
• The mitigation relies on a hand‑tuned λ hyperparameter; future work could learn this weighting automatically or integrate more sophisticated fairness‑aware objectives.
• Extending the framework to multi‑model ecosystems (e.g., ensembles of LLMs) and to online, streaming data scenarios is an open research direction.

Authors

• Yaxuan Wang
• Zhongteng Cai
• Yujia Bao
• Xueru Zhang
• Yang Liu

Paper Information

• arXiv ID: 2601.05184v1
• Categories: cs.AI, cs.CL
• Published: January 8, 2026
• PDF: Download PDF