[Paper] SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation

Source: arXiv - 2604.20842v1

Overview

The paper presents SpeechParaling‑Bench, a new benchmark designed to evaluate how well large audio‑language models (LALMs) can generate speech that conveys fine‑grained paralinguistic cues—things like emotion intensity, speaking style, and contextual adaptation. By expanding the feature set to over 100 nuanced attributes and introducing a scalable pairwise evaluation method, the authors expose significant gaps in current voice‑generation systems, even among top‑tier proprietary models.

Key Contributions

• Comprehensive feature taxonomy: Grows the number of evaluated paralinguistic dimensions from < 50 to > 100, covering static traits (e.g., pitch, timbre) and dynamic aspects (e.g., emotion shift within an utterance).
• Large multilingual query set: Provides > 1,000 English‑Chinese parallel speech prompts, enabling cross‑lingual assessment.
• Three‑tier task hierarchy:
  1. Fine‑grained control – static manipulation of individual cues.
  2. Intra‑utterance variation – dynamic modulation of cues within a single utterance.
  3. Context‑aware adaptation – adjusting speech to situational context or dialogue history.
• Pairwise comparison evaluation pipeline: Uses an LALM‑based judge to rank generated samples against a fixed baseline, turning subjective scoring into relative preference judgments and eliminating the need for costly human annotations.
• Empirical audit of state‑of‑the‑art LALMs: Demonstrates that even leading commercial models fail to reliably control or interpret a majority of paralinguistic features, with 43.3 % of dialogue errors traced to mis‑handled cues.

Methodology

1. Dataset Construction
   • Curated > 100 paralinguistic attributes (e.g., “whisper intensity”, “sarcasm level”).
   • Collected 1,000+ paired English‑Chinese speech queries, each annotated with target attribute values.
2. Task Design
   • Fine‑grained control: Models receive a single attribute specification and must synthesize speech matching it.
   • Intra‑utterance variation: Models are given a timeline of attribute changes (e.g., “start neutral, become excited after 2 s”).
   • Context‑aware adaptation: Models see preceding dialogue turns and must generate a response that aligns with both content and paralinguistic context.
3. Evaluation Pipeline
   • A pre‑trained LALM acts as a judge. For each test case, the judge receives two candidate outputs (one fixed baseline, one model under test) plus the original prompt.
   • The judge produces a binary preference (“A better than B”) based on how well each candidate satisfies the target paralinguistic profile.
   • Aggregating many pairwise votes yields a robust preference score, sidestepping absolute rating bias.

Results & Findings

• Static control: Top commercial models achieved only ~58 % preference over the baseline, indicating limited ability to hit exact attribute targets.
• Dynamic modulation: Performance dropped sharply (~42 % preference), revealing difficulty in handling intra‑utterance cue transitions.
• Contextual adaptation: Errors related to mis‑interpreting paralinguistic intent accounted for 43.3 % of dialogue failures, the largest error category across all tested systems.
• Baseline vs. human: Human‑rated samples still outperformed the best LALM by a wide margin, confirming a substantial quality gap.

Practical Implications

• Voice assistants & chatbots: Current assistants may sound “flat” or mis‑read user emotions, leading to awkward interactions. Improving paralinguistic control could make them sound more empathetic, persuasive, or culturally appropriate.
• Content creation tools: Podcasting, audiobooks, and game dialogue pipelines can benefit from fine‑grained style knobs, reducing the need for manual voice‑actor re‑recordings.
• Accessibility: Better modulation of prosody can aid screen‑readers for users with visual impairments, delivering information with clearer emphasis and emotional cues.
• Evaluation infrastructure: The pairwise LALM judge offers a low‑cost, scalable way for product teams to benchmark new TTS models without hiring large panels of annotators.

Limitations & Future Work

• Subjectivity of the judge: Although the pairwise approach reduces bias, it still inherits the LALM’s own preferences and may not capture all human nuances.
• Language scope: The benchmark currently focuses on English and Chinese; extending to more languages and dialects is needed for global applicability.
• Real‑world deployment: The study evaluates offline generation; integrating these controls into low‑latency, on‑device TTS pipelines remains an open challenge.
• Future directions: The authors suggest enriching the benchmark with multimodal context (e.g., video, facial expressions) and exploring reinforcement‑learning‑based fine‑tuning to close the gap between model outputs and human expectations.

Authors

• Ruohan Liu
• Shukang Yin
• Tao Wang
• Dong Zhang
• Weiji Zhuang
• Shuhuai Ren
• Ran He
• Caifeng Shan
• Chaoyou Fu

Paper Information

• arXiv ID: 2604.20842v1
• Categories: cs.CL, cs.AI, cs.SD
• Published: April 22, 2026
• PDF: Download PDF