[Paper] It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO
Source: arXiv - 2606.10931v1
Overview
Warning: This paper contains several toxic and offensive statements. Modern large language models (LLMs) are typically aligned through large-scale post-training to ensure fair and reliable behavior. In this work, we investigate how easily such guardrails can be broken by Group Relative Policy Optimization (GRPO). We show that one-shot GRPO training on a single biased example is sufficient to induce systematic bias, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. We further find that models differ in their susceptibility based on the initial likelihood of producing biased outputs. Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.
Key Contributions
This paper presents research in the following areas:
- cs.CL
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.CL.
Authors
- Naihao Deng
- Yilun Zhu
- Naichen Shi
- Clayton Scott
- Rada Mihalcea
Paper Information
- arXiv ID: 2606.10931v1
- Categories: cs.CL
- Published: June 9, 2026
- PDF: Download PDF