[Paper] LLMs as Strategic Actors: Behavioral Alignment, Risk Calibration, and Argumentation Framing in Geopolitical Simulations

Source: arXiv - 2603.02128v1

Overview

This paper investigates how today’s leading large language models (LLMs) behave when they are placed in the driver’s seat of geopolitical decision‑making simulations. By pitting six state‑of‑the‑art LLMs against human participants across four real‑world crisis scenarios, the authors assess whether the models can choose sensible actions, calibrate risk, and articulate their reasoning in a way that mirrors human diplomatic thinking.

Key Contributions

• Empirical benchmark: First large‑scale comparison of six popular LLMs with human players on multi‑round geopolitical crisis simulations.
• Behavioral alignment metrics: Introduces quantitative measures for action alignment, risk calibration (severity of chosen actions), and argumentative framing grounded in international‑relations theory.
• Temporal dynamics analysis: Shows how model behavior diverges from humans over successive simulation rounds, revealing distinct “strategic personalities.”
• Qualitative insight into LLM reasoning: Finds a consistent normative‑cooperative framing (stability, coordination, risk mitigation) across models, with minimal adversarial or power‑maximizing arguments.
• Open‑source artifacts: Releases the simulation scripts, prompt templates, and evaluation code for reproducibility and community extension.

Methodology

1. Simulation design – The authors adapted four well‑documented geopolitical crises (e.g., a border dispute, a resource embargo) into turn‑based decision games. Each round required a player to pick one of a predefined set of diplomatic or military actions and to provide a textual justification.
2. LLM selection & prompting – Six LLMs (including GPT‑4, Claude, Llama 2‑Chat, and others) were accessed via their public APIs. A uniform prompt template asked the model to (a) select an action, (b) explain the choice, and (c) cite relevant IR concepts (e.g., deterrence, balance of power).
3. Human baseline – 120 participants with varied expertise (policy analysts, graduate IR students, and hobbyists) played the same simulations under identical conditions.
4. Evaluation metrics
   • Action alignment: Jaccard similarity between model‑chosen actions and the human consensus per round.
   • Risk calibration: Mapping each action to a severity score (low‑medium‑high) and comparing the distribution to human risk profiles.
   • Argumentation framing: Automated text classification (using a fine‑tuned BERT model) to label explanations as normative‑cooperative, adversarial, or neutral, followed by manual validation.
5. Temporal analysis – Metrics were computed for each round to observe drift or convergence over the simulation horizon.

Results & Findings

• Early‑round alignment: In the first two rounds, all LLMs achieved >70 % Jaccard similarity with human actions, indicating they can capture the “baseline” diplomatic intuition.
• Divergence over time: By round 4, alignment dropped to 45 % for most models, while a few (e.g., GPT‑4) maintained a steadier 60 %—suggesting better strategic persistence.
• Risk calibration: Models tended to under‑estimate risk, selecting milder actions more often than humans, especially in high‑tension scenarios.
• Argumentation framing: >80 % of model explanations fell into the normative‑cooperative category, emphasizing stability and coordination. Adversarial framing (e.g., power projection, coercion) was rare (<5 %).
• Distinct behavioral profiles: Some models (Claude) displayed a “cautious” profile (low‑risk actions, frequent calls for negotiation), while others (Llama 2‑Chat) showed a “reactive” profile (quick escalation after a single adverse event).

Practical Implications

• Decision‑support prototypes: The findings suggest LLMs can serve as first‑pass advisors in crisis‑management tools, offering human‑like suggestions and rationales for low‑stakes or early‑stage analysis.
• Risk‑aware prompting: Developers must embed risk‑calibration prompts (e.g., “consider worst‑case consequences”) to counteract the models’ natural bias toward safe, cooperative actions.
• Simulation training: Game designers and policy‑training platforms can leverage LLM agents to generate diverse opponent strategies, enriching scenario variety without hiring subject‑matter experts.
• Explainability pipelines: The consistent normative framing can be harnessed for transparent AI‑assisted diplomacy dashboards, where the model’s justification is displayed alongside recommended actions.
• Compliance & governance: Since LLMs default to cooperative language, they may be less likely to produce aggressive policy recommendations, reducing the risk of unintended escalation in automated advisory systems.

Limitations & Future Work

• Scenario scope: Only four crises were tested; broader geopolitical contexts (e.g., cyber warfare, multi‑state coalitions) may expose different model behaviors.
• Prompt sensitivity: Results depend heavily on the prompt template; alternative phrasing could shift risk calibration or framing.
• Human baseline diversity: The human pool, while varied, lacked senior diplomatic practitioners, potentially skewing the “ground truth.”
• Evaluation granularity: The risk‑severity mapping is coarse; finer‑grained utility models could capture subtler strategic nuances.
• Future directions: The authors propose extending the benchmark to multi‑agent environments, integrating reinforcement‑learning fine‑tuning for strategic persistence, and exploring adversarial prompting to surface more diverse argumentative styles.

Authors

• Veronika Solopova
• Viktoria Skorik
• Maksym Tereshchenko
• Alina Haidun
• Ostap Vykhopen

Paper Information

• arXiv ID: 2603.02128v1
• Categories: cs.CL, cs.AI, cs.CY
• Published: March 2, 2026
• PDF: Download PDF