[Paper] Segmental Attention Decoding With Long Form Acoustic Encodings

Source: arXiv - 2512.14652v1

Overview

This paper tackles a long‑standing pain point for speech‑to‑text systems that rely on attention‑based encoder‑decoder (AED) models: they work well on short, neatly segmented utterances, but stumble when fed continuous, long‑form audio. The authors identify why the models lose their sense of “where they are” in a long stream and propose a set of practical fixes that let AED decoders operate autoregressively on unsegmented speech without sacrificing accuracy.

Key Contributions

• Diagnosed the root cause: AED models implicitly learn absolute frame positions from the limited context of training segments, which disappears in long‑form decoding, breaking the ordering of acoustic tokens.
• Explicit positional encodings in cross‑attention to restore absolute timing information for each decoded segment.
• Long‑form training regime that presents the model with extended acoustic contexts, forcing it to rely on true acoustic cues rather than segment‑boundary tricks.
• Segment concatenation strategy that randomly stitches together training segments, exposing the model to a wide variety of segmentation patterns.
• Semantic segmentation alignment that matches the decoder’s output segments to the natural linguistic boundaries used during training, improving consistency.
• Empirical validation showing the gap between continuous and segmented decoding disappears, enabling practical use of AED decoders on streaming audio.

Methodology

1. Baseline AED setup – The authors start with a standard transformer‑style encoder‑decoder trained on short utterances (e.g., 10‑second clips).
2. Problem analysis – They probe the model’s attention maps and discover that the cross‑attention keys/values become permutation‑invariant when the absolute position signal vanishes, causing the decoder to lose ordering information.
3. Four engineering interventions:
   • Positional injection: Add sinusoidal or learned absolute position vectors to the cross‑attention inputs for each segment the decoder processes.
   • Extended context training: During training, feed the encoder longer audio windows (up to several minutes) so the model can’t cheat by using segment‑edge cues.
   • Random concatenation: Randomly concatenate multiple training utterances to simulate diverse segment boundaries, preventing the model from over‑fitting to a single segmentation style.
   • Semantic segmentation: Use a downstream language model or forced alignment to define segment boundaries that correspond to meaningful linguistic units (e.g., sentences or phrases).
4. Evaluation – The modified system is tested on both artificially segmented audio and truly continuous recordings, measuring word error rate (WER) and decoding latency.

Results & Findings

Key takeaways

• Adding absolute positional encodings alone recovers most of the lost ordering, but the full suite of four tricks is needed to close the gap completely.
• The model retains its streaming capability: decoding latency grows linearly with segment length, not with the entire audio history.
• Qualitative analysis shows the decoder now produces coherent transcriptions across sentence boundaries, rather than “jumping” or repeating phrases.

Practical Implications

• Streaming ASR services (e.g., live captioning, voice assistants) can now adopt a single AED model for both short commands and long dictations, simplifying deployment pipelines.
• Reduced engineering overhead: No need to maintain separate models or hand‑crafted segmentation heuristics for different use‑cases.
• Improved user experience: More accurate, low‑latency transcription in scenarios like meetings, podcasts, or call‑center recordings where audio is naturally continuous.
• Compatibility with existing toolkits: The modifications are lightweight (positional embeddings, data augmentation) and can be plugged into popular frameworks like ESPnet, Fairseq, or Hugging Face Transformers.
• Potential for multimodal extensions: Since the approach restores temporal grounding, it could be combined with video or sensor streams where precise alignment matters.

Limitations & Future Work

• Scalability of long‑form training: Feeding several‑minute audio windows increases GPU memory usage; the authors suggest gradient checkpointing but more efficient architectures (e.g., memory‑compressed attention) could help.
• Dependence on quality of semantic segmentation: The alignment step assumes reasonably accurate forced alignments; noisy or low‑resource languages may struggle.
• Evaluation limited to English: Cross‑lingual robustness and performance on tonal or agglutinative languages remain open questions.
• Real‑time constraints: While latency is acceptable for near‑real‑time use, ultra‑low‑latency applications (e.g., live translation) may need further optimization.

Future research directions include exploring adaptive segment lengths based on acoustic confidence, integrating streaming‑friendly transformer variants, and extending the approach to end‑to‑end multilingual ASR systems.

Authors

• Pawel Swietojanski
• Xinwei Li
• Mingbin Xu
• Takaaki Hori
• Dogan Can
• Xiaodan Zhuang

Paper Information

• arXiv ID: 2512.14652v1
• Categories: eess.AS, cs.CL
• Published: December 16, 2025
• PDF: Download PDF