[Paper] Green Shielding: A User-Centric Approach Towards Trustworthy AI

Source: arXiv - 2604.24700v1

Overview

Large language models (LLMs) are being rolled out in high‑stakes settings such as medical decision support, but their answers can swing wildly just because users phrase the same question differently. The paper Green Shielding: A User‑Centric Approach Towards Trustworthy AI proposes a systematic way to study—and eventually mitigate—these “benign” variations, offering concrete guidance for safer deployments.

Key Contributions

• User‑centric evaluation framework (CUE): Defines benchmarks that combine realistic Context, clear Reference standards, and Utility‑focused metrics, together with Elicitation‑style perturbations that mimic everyday phrasing changes.
• HealthCareMagic‑Diagnosis (HCM‑Dx) benchmark: A curated set of patient‑written medical queries, complete with structured diagnosis reference sets and clinically meaningful evaluation metrics (e.g., coverage of critical conditions, plausibility of differential lists).
• Empirical analysis of prompt‑level factors: Shows how variations such as question framing, tone, or added context systematically shift LLM outputs along clinically relevant dimensions.
• Pareto‑style trade‑off discovery: Identifies a “neutralization” perturbation that strips away superficial user cues, yielding more concise, clinician‑like differentials but at the cost of missing some high‑risk diagnoses.
• Guidance for deployment: Demonstrates how the CUE criteria can be turned into actionable recommendations for developers building decision‑support tools in medicine and beyond.

Methodology

1. Benchmark Construction (CUE):

   • Context: Collected real‑world, patient‑authored queries from the HealthCareMagic platform.
   • Reference: Built structured diagnosis sets vetted by practicing physicians, covering both common and safety‑critical conditions.
   • Utility Metrics: Designed metrics that capture clinical usefulness:
     • Coverage – does the list include the true condition?
     • Plausibility – how medically reasonable are the suggested differentials?
     • Conciseness – length of the list.

2. Perturbation Design (Elicitation):

   • Created systematic variations of each query (e.g., adding/removing symptom detail, changing formality, reordering phrases).
   • Included a neutralization perturbation that removes user‑level stylistic cues while preserving core medical content.

3. Model Evaluation:

   • Tested several frontier LLMs (e.g., GPT‑4, Claude, LLaMA‑2) on the original and perturbed queries.
   • Measured how each perturbation moved the model’s output along the three utility axes, visualizing the results as Pareto frontiers.

4. Human Validation:

   • Physicians reviewed a sample of model‑generated differential lists to confirm that the automated metrics aligned with clinical judgment.

Results & Findings

• Prompt sensitivity is real: Even minor rephrasings caused noticeable shifts in diagnosis lists, sometimes swapping a life‑threatening condition for a benign one.
• Neutralization improves plausibility & brevity: Stripping away user‑level noise produced differential lists that clinicians rated as more realistic and easier to read.
• Trade‑off surface: The neutralized outputs covered fewer high‑risk conditions, highlighting a classic precision‑recall tension in safety‑critical AI.
• Pareto‑like behavior across models: All tested LLMs displayed similar trade‑off curves, suggesting the phenomenon is model‑agnostic rather than a quirk of a single architecture.

Practical Implications

• Deployment checklists: Teams can adopt the CUE criteria to audit their LLM‑powered tools before release, ensuring that benchmarks reflect real user language and clinical goals.
• Prompt‑design guidelines: UI/UX designers can embed “neutralization” steps (e.g., auto‑rephrasing user input) to improve answer quality while being aware of the coverage trade‑off.
• Risk‑aware monitoring: By tracking utility metrics in production (e.g., sudden drops in coverage for certain phrasing patterns), operators can trigger alerts or fallback to human review.
• Beyond healthcare: The same framework can be ported to legal advice, financial planning, or any decision‑support domain where user phrasing variability matters.

Limitations & Future Work

• Domain focus: The study is limited to medical diagnosis; other domains may exhibit different sensitivity patterns.
• Reference completeness: Even expert‑curated diagnosis sets can miss rare conditions, potentially biasing utility metrics.
• Scalability of perturbations: Generating exhaustive realistic variations for every possible user query remains computationally expensive.
• Future directions: Extending CUE to multimodal inputs (e.g., image‑plus‑text), automating perturbation generation with learned paraphrase models, and integrating real‑time user feedback loops to continuously refine the benchmark.

Authors

• Aaron J. Li
• Nicolas Sanchez
• Hao Huang
• Ruijiang Dong
• Jaskaran Bains
• Katrin Jaradeh
• Zhen Xiang
• Bo Li
• Feng Liu
• Aaron Kornblith
• Bin Yu

Paper Information

• arXiv ID: 2604.24700v1
• Categories: cs.CL, cs.AI
• Published: April 27, 2026
• PDF: Download PDF