[Paper] Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs

Source: arXiv - 2603.17902v1

Overview

Large language models (LLMs) are now being embedded as “AI agents” that can query internal company databases and produce context‑aware answers. While this boosts productivity, the generated text can unintentionally leak confidential data. Yang and Zhu propose a rigorous, differential‑privacy‑based framework that quantifies and controls this leakage from the enterprise data side—not just the user‑prompt side.

Key Contributions

• Probabilistic privacy model for AI agents – Treats the whole response generation pipeline (prompt + private dataset → token sequence) as a stochastic mechanism.
• Token‑level & message‑level differential privacy definitions – Extends classic DP to the granularity of individual tokens and whole messages, enabling fine‑grained leakage analysis.
• Closed‑form privacy bounds – Derives analytical relationships linking privacy loss to generation hyper‑parameters such as temperature, top‑k sampling, and output length.
• Privacy‑utility trade‑off formulation – Casts the choice of temperature (and related sampling knobs) as an optimization problem that minimizes privacy loss while preserving answer quality.
• Optimal temperature selection algorithm – Provides a practical recipe for picking the temperature that achieves the best privacy‑utility balance under a given DP budget.

Methodology

1. Stochastic Mechanism Definition – The authors model the AI agent as a function M(prompt, D) → distribution over token strings, where D is the private enterprise dataset.

2. Differential Privacy Adaptation – They adapt the classic (ε,δ)‑DP definition to two levels:

   • Token‑level DP: Guarantees that the probability of any single token changes only slightly when a single record in D is added/removed.
   • Message‑level DP: Extends the guarantee to the entire generated response.

3. Bounding Privacy Loss – By analyzing the softmax sampling step of LLMs, they express the privacy loss ε as a function of temperature τ, vocabulary size, and the number of generated tokens L.

4. Optimization Problem – They formulate:

   [ \min_{\tau} ; \text{Utility}(\tau) \quad \text{s.t.} \quad \epsilon(\tau, L) \le \epsilon_{\text{budget}} ]

   where utility is measured by standard language‑model metrics (e.g., perplexity, BLEU).

5. Solution via Convex Approximation – The paper shows the objective is quasi‑convex in τ, allowing efficient line‑search to find the optimal temperature.

Results & Findings

• Temperature matters: Lower temperatures (more deterministic sampling) dramatically reduce ε but hurt fluency; higher temperatures increase leakage.
• Token‑level DP is tighter: Guarantees at token granularity give smaller ε than naïve message‑level bounds.
• Optimal τ yields ≈30 % privacy improvement over default settings (τ = 1.0) with < 5 % utility loss.

The authors also validate the theoretical bounds empirically, showing the derived ε closely matches observed privacy leakage measured via membership inference attacks.

Practical Implications

• Enterprise AI agents can be safely deployed: By tuning temperature (or analogous sampling knobs) according to the provided formulas, engineers can meet regulatory privacy budgets (e.g., GDPR‑style DP guarantees) without sacrificing answer relevance.
• Built‑in privacy controls for LLM APIs: Cloud providers could expose a “privacy‑aware temperature” slider that automatically enforces the optimal τ for a given ε budget.
• Compliance‑first prompt engineering: Teams can now quantify how much sensitive information might leak from a generated response and adjust generation parameters or add post‑processing (e.g., redaction) accordingly.
• Guidance for fine‑tuning: When fine‑tuning on proprietary data, the framework helps decide how much to lower temperature or truncate responses to stay within a desired privacy envelope.

Limitations & Future Work

• Assumes static datasets: The analysis treats the private corpus as fixed; continual learning or dynamic updates could invalidate the bounds.
• Focus on temperature only: Other generation knobs (top‑p, nucleus sampling, beam width) are not fully explored, though they also affect privacy.
• Utility metric simplification: Perplexity and BLEU are proxies; real‑world task performance (e.g., decision‑support accuracy) may behave differently.
• Scalability to massive models: Experiments were run on 2.7 B‑parameter models; extending to 100 B‑scale LLMs may require additional approximations.

Future research directions include extending the DP analysis to streaming query workloads, incorporating other sampling strategies, and building end‑to‑end toolkits that automatically enforce the optimal privacy‑utility trade‑off in production AI‑agent pipelines.

Authors

• Ya‑Ting Yang
• Quanyan Zhu

Paper Information

• arXiv ID: 2603.17902v1
• Categories: cs.CR, cs.AI
• Published: March 18, 2026
• PDF: Download PDF