[Paper] Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition

Source: arXiv - 2606.07309v1

Overview

Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question in speech emotion recognition (SER) by deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set. These tokens summarise energy, pitch, dynamics, brightness, formants, and voice quality, and are appended to the textual prompt while the audio input is kept unchanged. Across the widely used FAU-Aibo and IEMOCAP benchmarks, aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance relative to aligned tokens and shift confusions toward neutral. Importantly, predictions do not collapse under strong token perturbations, suggesting that the models are sensitive to the symbolic cue channel but remain partly anchored to the audio signal. We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.

Key Contributions

This paper presents research in the following areas:

• cs.SD
• cs.AI
• cs.CL

Methodology

Please refer to the full paper for detailed methodology.

Practical Implications

This research contributes to the advancement of cs.SD.

Authors

• Iosif Tsangko
• Andreas Triantafyllopoulos
• Björn W. Schuller

Paper Information

• arXiv ID: 2606.07309v1
• Categories: cs.SD, cs.AI, cs.CL
• Published: June 5, 2026
• PDF: Download PDF