[Paper] Detecting Winning Arguments with Large Language Models and Persuasion Strategies

Source: arXiv - 2601.10660v1

Overview

The paper explores how large language models (LLMs) can be taught to spot “winning” arguments by explicitly reasoning about six classic persuasion tactics (e.g., attacking reputation, distraction, manipulative wording). By embedding these strategies into the prompting process, the authors achieve more accurate and interpretable predictions of argument persuasiveness across three real‑world datasets, including the popular Change My View subreddit corpus.

Key Contributions

• Strategy‑guided prompting: Introduces a Multi‑Strategy Persuasion Scoring (MSPS) framework that steers LLMs to evaluate arguments through the lens of six predefined persuasion strategies.
• Cross‑dataset validation: Demonstrates the approach on three diverse, publicly‑available argument corpora (Winning Arguments, Anthropic/Persuasion, Persuasion for Good), showing consistent gains over baseline LLM prompts.
• Topic‑annotated Winning Arguments: Releases a new version of the Change My View dataset enriched with high‑level discussion topics, enabling finer‑grained analysis of strategy effectiveness.
• Interpretability boost: Shows that strategy‑level scores provide transparent rationales for why an argument is deemed persuasive, facilitating debugging and trust.
• Robustness insights: Empirically finds that strategy‑aware prompting reduces performance variance across topics and mitigates over‑reliance on surface lexical cues.

Methodology

1. Persuasion strategy taxonomy – The authors adopt six well‑studied tactics (Attack on Reputation, Distraction, Manipulative Wording, etc.) drawn from rhetorical theory.
2. Prompt engineering – For each argument, the LLM receives a structured prompt that asks it to (a) identify evidence of each strategy, (b) assign a confidence score (0–1) per strategy, and (c) aggregate these scores into an overall persuasiveness prediction.
3. Model backbone – Experiments use GPT‑3.5‑turbo and Claude‑2 as the underlying LLMs, accessed via API with temperature set to 0 for deterministic outputs.
4. Training‑free fine‑tuning – No gradient‑based fine‑tuning is performed; the method relies purely on prompt design and few‑shot examples that illustrate each strategy.
5. Evaluation – Standard classification metrics (accuracy, F1) compare the MSPS approach against a vanilla LLM baseline (single‑prompt “Is this argument persuasive?”) and a supervised BERT‑style classifier trained on the same data.
6. Topic analysis – The enriched Winning Arguments dataset is split by discussion topic (e.g., politics, technology, health) to assess whether certain strategies dominate in specific domains.

Results & Findings

• Strategy scores matter: Ablation where one or more strategies are omitted drops accuracy by 3–5 %, confirming each tactic contributes useful signal.
• Topic robustness: Accuracy variance across topics shrinks from ±6 % (baseline) to ±3 % (MSPS), indicating the model is less swayed by topic‑specific vocabulary.
• Interpretability: Human evaluators rated the strategy‑level explanations as “clear” in 84 % of cases, compared to 57 % for the vanilla LLM output.

Practical Implications

• Content moderation & fact‑checking: Platforms can flag potentially manipulative posts by scoring them on the six strategies, enabling more nuanced policy enforcement than simple toxicity filters.
• Automated debate assistants: Developers building AI coaches for public speaking or negotiation can surface which persuasion tactics are strongest in a draft argument, offering targeted suggestions for improvement.
• Marketing & copywriting tools: Strategy‑aware scoring can help marketers audit ad copy for overly aggressive or deceptive tactics, aligning with regulatory compliance (e.g., FTC guidelines).
• Educational tech: Argument‑analysis modules in MOOCs or writing platforms can provide students with transparent feedback on the rhetorical moves they employ, fostering critical thinking.
• Dataset enrichment: The released topic‑annotated Winning Arguments set can serve as a benchmark for future work on domain‑aware persuasion detection, encouraging reproducibility and community contributions.

Limitations & Future Work

• Prompt sensitivity: Results depend on careful prompt phrasing and few‑shot examples; small changes can affect scores, suggesting a need for more robust prompting pipelines.
• Strategy coverage: The six‑strategy taxonomy, while grounded in literature, may miss nuanced tactics (e.g., emotional appeals) that appear in specialized domains.
• LLM bias: Since the approach leans on proprietary LLMs, any inherent biases (e.g., cultural or political) could influence strategy detection, warranting bias audits.
• Scalability: Per‑argument multi‑step prompting incurs higher API costs and latency; future work could explore distilled models that internalize the strategy reasoning.
• Cross‑lingual extension: All experiments are English‑only; extending the framework to multilingual settings would broaden applicability in global platforms.

Authors

• Tiziano Labruna
• Arkadiusz Modzelewski
• Giorgio Satta
• Giovanni Da San Martino

Paper Information

• arXiv ID: 2601.10660v1
• Categories: cs.CL
• Published: January 15, 2026
• PDF: Download PDF