[Paper] CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

Source: arXiv - 2604.15267v1

Overview

The paper CoopEval investigates why today’s most capable large‑language‑model (LLM) agents tend to defect rather than cooperate in classic “social dilemma” games such as the Prisoner’s Dilemma or public‑goods games. By systematically testing four game‑theoretic mechanisms that are known to foster cooperation among rational agents, the authors identify which of these mechanisms actually work when the players are LLM‑driven bots. Their findings have immediate safety relevance for any product that lets LLM agents negotiate, trade, or collaborate with humans or other AI systems.

Key Contributions

• First systematic benchmark of cooperation‑sustaining mechanisms (repetition, reputation, mediation, contracting) on modern LLM agents.
• Empirical evidence that state‑of‑the‑art LLMs (with or without chain‑of‑thought prompting) default to defection in one‑shot dilemmas.
• Discovery that contracting and third‑party mediation are the most reliable levers for achieving stable cooperation among capable models.
• Analysis of robustness showing that repetition‑based cooperation collapses when opponent behavior varies across rounds.
• Evolutionary‑pressure experiments demonstrating that the same mechanisms become even more effective when agents are trained to maximize long‑term payoffs.

Methodology

1. Social‑dilemma suite – Four canonical games were implemented:

   • (a) Prisoner’s Dilemma
   • (b) Public Goods
   • (c) Stag Hunt
   • (d) A multi‑player resource‑allocation game.
     Each game captures a different facet of “robust cooperation” (e.g., risk of exploitation, need for coordination).

2. LLM agents – Several recent LLM families were used (GPT‑4, Claude‑2, Llama‑2‑70B, etc.) and evaluated both with standard prompting and with chain‑of‑thought (CoT) reasoning prompts.

3. Cooperation mechanisms – For each game the authors instantiated:

   • Repeated interaction (iterated game with discounting).
   • Reputation system (public score visible to the opponent).
   • Third‑party mediator (a neutral AI that decides the joint action).
   • Contractual agreement (pre‑committed conditional payments).

4. Evaluation protocol – Agents played thousands of matches under each mechanism. The authors recorded the frequency of cooperative outcomes, average payoffs, and stability across opponent variations.

5. Evolutionary pressure test – Agents were fine‑tuned with a simple reinforcement‑learning loop that rewarded higher cumulative payoffs, to see how the mechanisms behave when agents adapt over time.

Results & Findings

• Defection dominance: In single‑shot games, all tested LLMs chose to defect >90 % of the time, regardless of prompting style.
• Contracting & mediation: These mechanisms effectively align incentives, turning the game into a coordination problem rather than a conflict.
• Evolutionary boost: When agents were fine‑tuned to maximize long‑term payoff, cooperation under contracting rose to >95 %, and mediation remained >90 %. Repetition’s cooperation rate improved only modestly (~75 %).

Practical Implications

• Designing safe multi‑agent systems – Embedding a contractual layer (smart‑contract style escrow or conditional payment) is far more reliable than relying on repeated interactions or reputation alone for platforms where LLM bots negotiate contracts (e.g., automated procurement, decentralized finance, collaborative coding assistants).
• Third‑party arbitration services – Deploying a neutral “mediator” LLM that decides joint actions can act as a safety net for peer‑to‑peer AI marketplaces, reducing the risk of exploitative behavior.
• Prompt engineering guidelines – Simple chain‑of‑thought prompts do not magically induce cooperation; developers should focus on structural incentives.
• Regulatory & compliance tools – The benchmark provides a concrete methodology for auditors to test whether AI agents in a given ecosystem are likely to cooperate under defined rules, supporting compliance with emerging AI‑safety standards.
• Evolutionary fine‑tuning – Training LLM agents with long‑term payoff objectives (e.g., via RLHF) can amplify the benefits of contracts and mediation, suggesting a path for “cooperative AI” product pipelines.

Limitations & Future Work

• Model scope – The study examined a limited set of publicly available LLMs; proprietary or smaller fine‑tuned models may behave differently.
• Simplified game settings – Real‑world negotiations involve richer action spaces, asymmetric information, and external enforcement costs that are not captured by the abstract games used here.
• Mediator trust assumptions – The paper assumes the mediator is trusted and unbiased; future work should explore mechanisms for verifying mediator integrity.
• Scalability of contracts – Implementing enforceable contracts at scale (e.g., on blockchain) introduces latency and cost considerations that were not evaluated.
• Long‑term dynamics – While evolutionary pressure experiments show promising trends, longer‑horizon simulations with heterogeneous populations remain an open research direction.

CoopEval offers a practical roadmap for developers who need their LLM agents to play nicely with one another—and with humans—by showing that structured, enforceable agreements and neutral arbitration are the most effective levers for fostering cooperation. Incorporating these insights early can help avoid costly safety pitfalls as AI agents become more autonomous and economically influential.

Authors

• Emanuel Tewolde
• Xiao Zhang
• David Guzman Piedrahita
• Vincent Conitzer
• Zhijing Jin

Paper Information

• arXiv ID: 2604.15267v1
• Categories: cs.GT, cs.AI, cs.CL, cs.CY, cs.MA
• Published: April 16, 2026
• PDF: Download PDF