[Paper] MonarchRT: Efficient Attention for Real-Time Video Generation

Source: arXiv - 2602.12271v1

Overview

MonarchRT tackles the biggest bottleneck in real‑time video generation with diffusion transformers: the quadratic cost of 3‑D self‑attention. By redesigning the attention mechanism to be both highly expressive and sparsely computed, the authors enable true real‑time video synthesis (≈16 FPS) on a single consumer‑grade GPU, a milestone for interactive AI‑driven media.

Key Contributions

• Monarch‑RT attention: a novel structured‑sparsity scheme based on Monarch matrices that captures periodic spatiotemporal patterns and dynamic semantic correspondences while keeping computation cheap.
• Extended tiled Monarch parameterization: aligns block structures with video dimensions, delivering up to 95 % attention sparsity with no perceptual quality loss.
• Custom Triton kernels: hand‑optimized GPU kernels that make the new attention faster than FlashAttention‑2/3/4 on high‑end GPUs (RTX 5090, H100, B200).
• Empirical validation: demonstrates that Monarch‑RT outperforms existing sparse‑attention baselines on the state‑of‑the‑art Self‑Forcing diffusion model, achieving 1.4–11.8× speed‑ups and real‑time 16 FPS video generation.
• Open‑source‑ready implementation: the authors release the Triton kernels and integration code, lowering the barrier for developers to adopt the technique.

Methodology

1. Problem Insight – In few‑step, autoregressive video diffusion, attention is not purely sparse; it mixes three components:

   • Periodic positional structure (regular motion patterns)
   • Dynamic sparse semantic links (objects that appear/disappear)
   • Dense local mixing (pixel‑level texture continuity)

2. Monarch Matrix Factorization – The authors decompose the full attention matrix into a set of aligned blocks (Monarch blocks) that respect the video’s spatiotemporal grid. Each block is either:

   • Dense (for local mixing) or
   • Low‑rank / top‑k (for long‑range semantic links).

3. Extended Tiling – By tiling the Monarch blocks across time and space, the scheme captures periodic patterns without needing a full‑size attention map.

4. Parameterization & Finetuning – The block structure is learned as a set of lightweight parameters. A short finetuning stage on the target diffusion model (Self‑Forcing) adapts these parameters without expensive retraining.

5. GPU Acceleration – Custom Triton kernels execute the block‑wise attention efficiently, bypassing the memory‑bandwidth limits of generic kernels like FlashAttention.

Results & Findings

• Monarch‑RT matches or slightly improves visual quality (FID, perceptual metrics) while delivering order‑of‑magnitude speed gains.
• The method remains robust across resolutions (64×64 up to 256×256) and different hardware generations.

Practical Implications

• Interactive Media Creation – Game developers, VFX artists, and AR/VR creators can now generate on‑the‑fly video assets (e.g., character animations, background loops) without pre‑rendering.
• Low‑Latency AI Services – Cloud providers can offer real‑time video synthesis APIs at reduced GPU cost, making pricing competitive.
• Edge Deployment – The high sparsity and custom kernels lower memory footprints, opening the door for real‑time diffusion video on high‑end laptops or future AI accelerators.
• Research Acceleration – By providing a plug‑and‑play attention module, researchers can experiment with diffusion video models without being bottlenecked by attention costs.

Limitations & Future Work

• Hardware Specificity – The current speedups rely on Nvidia GPUs and Triton; porting to other architectures (AMD, Apple Silicon) will need new kernels.
• Model Compatibility – Monarch‑RT was evaluated primarily with Self‑Forcing; adapting it to other diffusion backbones may require additional finetuning.
• Temporal Horizon – Extremely long video sequences (>10 s) could still hit memory limits due to the tiled block layout; future work may explore hierarchical or recurrent extensions.

MonarchRT marks a decisive step toward making diffusion‑based video generation practical for real‑time applications, bridging the gap between cutting‑edge research and production‑ready tools.

Authors

• Krish Agarwal
• Zhuoming Chen
• Cheng Luo
• Yongqi Chen
• Haizhong Zheng
• Xun Huang
• Atri Rudra
• Beidi Chen

Paper Information

• arXiv ID: 2602.12271v1
• Categories: cs.CV, cs.LG
• Published: February 12, 2026
• PDF: Download PDF