[Paper] Diffusion Forcing for Multi-Agent Interaction Sequence Modeling

Source: arXiv - 2512.17900v1

Overview

The paper presents MAGNet (Multi‑Agent Diffusion Forcing Transformer), a single neural model that can generate realistic motion for any number of interacting people. By marrying diffusion‑based generative modeling with a transformer that explicitly reasons about how agents influence each other, MAGNet can handle tasks ranging from predicting a partner’s next move to synthesizing whole group performances that span hundreds of frames.

Key Contributions

• Unified autoregressive diffusion framework for multi‑agent motion generation, eliminating the need for task‑specific models.
• Dyadic prediction, partner inpainting, and full‑scene generation all supported by the same architecture.
• Explicit inter‑agent coupling during the denoising steps, enabling coherent coordination across agents of arbitrary group size.
• Scalable design that is agnostic to the number of participants, allowing seamless extension from two‑person (dyadic) to three‑plus (polyadic) interactions.
• Ultra‑long sequence generation (hundreds of timesteps) while preserving temporal consistency and spatial plausibility.

Methodology

1. Diffusion Forcing Backbone – The model treats motion generation as a reverse diffusion process: it starts from random noise and iteratively “denoises” it to produce a plausible motion trajectory.
2. Transformer‑Based Conditioning – At each denoising step, a transformer encoder ingests the partially generated poses of all agents together with any external conditioning (e.g., a target activity label or a partial observation).
3. Inter‑Agent Coupling Layer – A dedicated attention module computes pairwise interactions between agents, ensuring that the update for one agent’s pose is informed by the current poses of its partners. This is the core “diffusion forcing” that drives coordinated behavior.
4. Autoregressive Sampling – The model generates frames sequentially: after producing frame t, it conditions the next diffusion step on the newly generated poses, allowing the system to maintain long‑range temporal coherence.
5. Flexible Conditioning – By swapping in different conditioning signals (e.g., a single agent’s observed motion, a high‑level activity tag, or no conditioning at all), the same network can perform prediction, inpainting, or free‑form generation.

Results & Findings

• Dyadic Benchmarks – On standard two‑person interaction datasets (e.g., dancing, boxing), MAGNet matches or slightly exceeds the performance of specialized state‑of‑the‑art models in terms of pose error and visual realism.
• Polyadic Scenarios – In experiments with three‑plus agents, MAGNet maintains tight synchronization (e.g., group dance formations) and realistic spacing, outperforming baseline methods that were originally designed for only two agents.
• Long‑Horizon Generation – The model successfully generates coherent motion for sequences up to 300 frames, with minimal drift or collapse, a notable improvement over prior diffusion‑based motion generators that struggled beyond ~50 frames.
• Ablation Studies – Removing the inter‑agent coupling layer leads to noticeable desynchronization, confirming that explicit interaction modeling is crucial for coordinated behavior.

Practical Implications

• Robotics & Human‑Robot Collaboration – MAGNet can be used to predict human teammates’ motions in real time, enabling robots to adapt their trajectories for safe, fluid cooperation in manufacturing or assistive settings.
• Virtual Production & Gaming – Content creators can generate crowd or group animations on‑the‑fly without hand‑crafting each character’s motion, dramatically reducing production time for movies, VR experiences, and multiplayer games.
• Social Computing & Telepresence – Real‑time synthesis of plausible group gestures can enrich remote collaboration tools, making avatars appear more natural during meetings or virtual events.
• Data Augmentation – Synthetic multi‑person motion can supplement scarce labeled datasets for downstream tasks like action recognition, pose estimation, or behavior prediction.

Limitations & Future Work

• Computational Cost – Autoregressive diffusion requires multiple denoising passes per frame, which can be expensive for real‑time applications; the authors suggest exploring accelerated sampling or distilled models.
• Dependence on High‑Quality Pose Data – Training relies on clean 3D pose annotations; noisy or occluded inputs may degrade performance.
• Limited Semantic Control – While activity labels can steer generation, fine‑grained control (e.g., specifying exact trajectories or interpersonal distances) remains an open challenge.
• Future Directions – The authors propose integrating physics‑based constraints, extending the framework to heterogeneous agents (e.g., humans + robots), and investigating hierarchical diffusion schemes to further speed up long‑sequence generation.

Authors

• Vongani H. Maluleke
• Kie Horiuchi
• Lea Wilken
• Evonne Ng
• Jitendra Malik
• Angjoo Kanazawa

Paper Information

• arXiv ID: 2512.17900v1
• Categories: cs.CV, cs.RO
• Published: December 19, 2025
• PDF: Download PDF