[Paper] Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework

Source: arXiv - 2512.16816v1

Overview

Large Language Models (LLMs) are now core building blocks of everything from chatbots to code assistants, but their decisions can unintentionally reflect societal biases. The paper introduces CAFFE (Counterfactual Assessment Framework for Fairness Evaluation), a systematic, intent‑aware testing harness that lets engineers probe LLMs for counterfactual fairness—i.e., whether a model would give the same answer if protected attributes (gender, race, etc.) were swapped.

Key Contributions

• Formal test‑case model that captures prompt intent, conversational context, input variants, fairness thresholds, and environment settings.
• Automated test‑data generation that creates realistic counterfactual variants (e.g., “John” ↔ “Jane”, “engineer” ↔ “nurse”).
• Semantic similarity‑based oracle to compare model responses while tolerating harmless wording changes.
• Empirical evaluation on three LLM families (decoder‑only, encoder‑decoder, and instruction‑tuned) showing higher bias coverage than prior metamorphic testing techniques.
• Open‑source prototype and a reusable test‑suite that can be plugged into CI pipelines.

Methodology

1. Test‑case Specification – Test writers declare a scenario (e.g., “recommend a candidate for a software role”) and list the protected attributes to vary.
2. Variant Generation – CAFFE leverages a lexical resource and a small LLM prompt to synthesize counterfactual inputs (e.g., swapping gendered names or pronouns).
3. Execution Engine – The original and each variant are sent to the target LLM under identical temperature, max‑tokens, and system‑prompt settings.
4. Fairness Oracle – Responses are embedded with a state‑of‑the‑art sentence encoder (e.g., SBERT). Pairwise cosine similarity is compared against a configurable threshold; a drop below the threshold flags a potential fairness violation.
5. Reporting – Violations are aggregated by attribute, intent, and model version, producing a concise dashboard for developers.

The workflow mirrors classic non‑functional testing (e.g., performance or security testing) but is tuned to the linguistic nature of LLM outputs.

Results & Findings

• Broader coverage: CAFFE discovered fairness issues in 15–20 % more attribute‑intent combos than the leading metamorphic baseline.
• More reliable detection: By using semantic similarity rather than exact string matching, the framework reduced spurious failures caused by harmless rephrasings.
• Scalability: Generating and evaluating 1,000+ test cases took under 30 minutes on a single GPU, making it feasible for CI integration.

Practical Implications

• CI/CD Ready: Teams can embed CAFFE into automated test suites, catching bias regressions before a model ships to production.
• Regulatory Alignment: The explicit fairness thresholds and audit trail help satisfy emerging AI governance standards (e.g., EU AI Act).
• Product Design: By surfacing which intents are most vulnerable (e.g., hiring, loan advice), product managers can prioritize mitigation strategies such as prompt engineering, fine‑tuning, or post‑processing filters.
• Cross‑Model Benchmarking: The framework’s neutral oracle lets engineers compare fairness across different LLM providers without hand‑crafting bespoke prompts for each.

Limitations & Future Work

• Semantic Oracle Sensitivity: Cosine similarity may still conflate subtle bias with legitimate content shifts; calibrating thresholds per domain remains manual.
• Attribute Scope: Current variant generation focuses on binary gender and a handful of ethnicity markers; extending to intersectional and non‑binary attributes is an open challenge.
• Context Length: Very long conversational histories can exceed model context windows, limiting the framework’s applicability to multi‑turn dialogues.
• Future Directions: The authors plan to (1) integrate causal inference techniques for deeper counterfactual reasoning, (2) expand the lexical resource with community‑curated bias lexicons, and (3) open an online leaderboard for cross‑organization fairness benchmarking.

Authors

• Alessandra Parziale
• Gianmario Voria
• Valeria Pontillo
• Gemma Catolino
• Andrea De Lucia
• Fabio Palomba

Paper Information

• arXiv ID: 2512.16816v1
• Categories: cs.SE
• Published: December 18, 2025
• PDF: Download PDF