[Paper] You Never Know a Person, You Only Know Their Defenses: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations

Source: arXiv - 2512.15601v1

Overview

This paper presents PsyDefConv, the first publicly‑available dialogue dataset annotated for psychological defense mechanisms—the unconscious strategies people use to protect themselves from emotional distress. By pairing the corpus with an evidence‑driven annotation assistant (DMRS Co‑Pilot), the authors show how to streamline the notoriously hard task of labeling defenses in real‑world supportive conversations, opening the door for AI systems that can sense and respond to a speaker’s defensive posture.

Key Contributions

• PsyDefConv corpus: 200 multi‑turn supportive dialogues (≈4.7 k utterances) with fine‑grained labels for four defense levels (immature, neurotic, mature, none).
• DMRS Co‑Pilot: a four‑stage, evidence‑backed annotation pipeline that cuts human labeling time by 22 % while maintaining solid inter‑annotator agreement (Cohen’s κ = 0.639).
• Comprehensive benchmark: zero‑shot and fine‑tuned experiments with state‑of‑the‑art language models (e.g., GPT‑3.5, LLaMA‑2) achieving a best macro F1 of ≈30 %, highlighting ample room for improvement.
• Empirical insights: mature defenses dominate the dataset; emotion‑specific patterns (e.g., higher immature defenses in anger‑laden turns) are uncovered.
• Open resources: full dataset, annotation code, and prompt templates released for reproducibility and community extension.

Methodology

1. Data collection – 200 real‑world supportive conversations were sourced from mental‑health chat platforms, anonymized, and split into speaker (help‑seeker) and responder turns.
2. Label schema – Drawing from psychotherapy theory, each help‑seeker utterance was tagged with one of four categories: None, Immature, Neurotic, or Mature defenses.
3. DMRS Co‑Pilot pipeline
   • Stage 1: Automatic cue extraction (lexical triggers, sentiment shifts).
   • Stage 2: Retrieval of relevant psychological literature snippets (evidence base).
   • Stage 3: Prompt generation for a large language model to propose a defense label with justification.
   • Stage 4: Human annotator review and final decision.
4. Annotation study – A counterbalanced experiment compared pure manual labeling vs. Co‑Pilot‑assisted labeling, measuring time, agreement, and perceived usefulness on a 7‑point Likert scale.
5. Model benchmarking – Zero‑shot prompting and supervised fine‑tuning were applied to several LLMs; performance was evaluated with macro F1 and confusion analysis.

Results & Findings

• Model performance: Best fine‑tuned LLaMA‑2‑7B achieved macro F1 ≈ 30 %; all models tended to over‑predict mature defenses.
• Corpus analysis: ~55 % of help‑seeker turns exhibited mature defenses, ~20 % neurotic, ~15 % immature, and ~10 % none. Emotion‑specific deviations (e.g., higher immature defenses during sadness) were statistically significant.

Practical Implications

• Enhanced conversational agents – Chatbots for mental‑health support can be equipped to detect defensive stances, allowing them to adapt phrasing (e.g., using more reflective listening when immature defenses surface).
• Therapist‑in‑the‑loop tools – Automated pre‑annotation can speed up charting of session transcripts, helping clinicians spot patterns of over‑reliance on certain defenses across sessions.
• Risk triage – Early detection of rigid, immature defenses may flag users at higher risk of disengagement or worsening symptoms, prompting timely human escalation.
• Dataset for transfer learning – PsyDefConv can serve as a downstream fine‑tuning set for any LLM aimed at mental‑health NLP, improving domain‑specific sensitivity without needing massive proprietary corpora.
• Explainability – The Co‑Pilot’s evidence‑backed suggestions provide a transparent rationale that aligns with clinical reasoning, easing regulatory and ethical concerns for AI‑assisted care.

Limitations & Future Work

• Scope of defenses – The four‑level taxonomy, while grounded in theory, may oversimplify the nuanced spectrum of defensive behavior.
• Domain bias – All dialogues come from text‑based supportive platforms; voice or multimodal cues (tone, facial expression) are absent.
• Model ceiling – Current LLMs still struggle (≈30 % F1), indicating a need for richer contextual modeling or hybrid rule‑based approaches.
• Generalizability – The dataset size (200 dialogues) is modest; scaling up and diversifying sources (different cultures, languages) will be crucial.

Future research directions include expanding the corpus, integrating multimodal signals, exploring few‑shot prompting strategies, and evaluating downstream impact on real‑world mental‑health chatbot deployments.

Authors

• Hongbin Na
• Zimu Wang
• Zhaoming Chen
• Peilin Zhou
• Yining Hua
• Grace Ziqi Zhou
• Haiyang Zhang
• Tao Shen
• Wei Wang
• John Torous
• Shaoxiong Ji
• Ling Chen

Paper Information

• arXiv ID: 2512.15601v1
• Categories: cs.CL
• Published: December 17, 2025
• PDF: Download PDF