[Paper] World-R1: Reinforcing 3D Constraints for Text-to-Video Generation

Source: arXiv - 2604.24764v1

Overview

World‑R1 tackles a persistent problem in text‑to‑video generation: the output often looks good frame‑by‑frame but falls apart when you look at the scene’s 3‑D geometry over time. By treating geometric consistency as a reinforcement‑learning (RL) objective, the authors boost 3‑D coherence without touching the underlying diffusion architecture, making the method lightweight enough to plug into existing video foundation models.

Key Contributions

• RL‑based 3‑D constraint enforcement – Introduces a reinforcement learning loop (Flow‑GRPO) that rewards videos for matching the spatial structure predicted by pre‑trained 3‑D foundation models.
• Pure‑text world‑simulation dataset – Curates a large, text‑only corpus describing static and dynamic 3‑D scenes, enabling the model to learn world‑level constraints from language alone.
• Architecture‑agnostic fine‑tuning – Improves geometric consistency while preserving the original visual fidelity, avoiding costly redesigns of the diffusion backbone.
• Periodic decoupled training schedule – Alternates between “rigid” (geometry‑focused) and “fluid” (motion‑focused) training phases, striking a balance between structural stability and natural motion.
• Comprehensive evaluation – Shows measurable gains in 3‑D consistency metrics and human preference studies across several benchmark video generation tasks.

Methodology

1. Base Model – Starts from a state‑of‑the‑art text‑to‑video diffusion model (e.g., Imagen‑Video, Make‑A‑Video).
2. 3‑D Feedback Sources
   • 3‑D foundation model: a pretrained neural renderer that predicts depth, pose, and mesh from video frames.
   • Vision‑language model: CLIP‑like encoder that scores how well the generated frames align with the input prompt.
3. Reinforcement Loop (Flow‑GRPO)
   • The video generator proposes a short clip.
   • The 3‑D model extracts geometric descriptors (depth maps, camera trajectories).
   • A reward function combines geometric consistency (e.g., low depth variance across frames) and semantic relevance (CLIP similarity).
   • Policy Gradient with Reward‑Based Proximal Optimization (GRPO) updates the generator’s parameters.
4. Training Schedule
   • Rigid Phase (every N steps): high weight on geometry reward → forces the model to respect static structures.
   • Fluid Phase: lower geometry weight, higher motion/texture reward → restores natural dynamics.
5. Dataset – The “World‑Sim” corpus contains ~200 k textual scene descriptions (e.g., “a marble statue rotates slowly in a sunlit atrium”) that explicitly encode 3‑D relationships, enabling the RL agent to learn from language alone.

Results & Findings

• Geometric consistency improves by ~45 % on average, cutting jitter and depth drift.
• Visual quality (sharpness, color fidelity) remains on par with the original model, confirming the “architecture‑agnostic” claim.
• Qualitative examples show stable objects (e.g., a rotating cube) that maintain shape across dozens of frames, something baseline models often lose after a few seconds.

Practical Implications

Limitations & Future Work

• Reward design complexity – Balancing geometry vs. motion rewards requires careful tuning; suboptimal weights can lead to overly stiff or overly fluid videos.
• Dependence on 3‑D priors – The quality of the external 3‑D foundation model directly caps the achievable consistency; errors in depth estimation propagate to the generator.
• Scalability to long videos – Experiments focus on clips ≤ 8 seconds; extending to minute‑scale narratives may need hierarchical RL or memory mechanisms.
• Dataset bias – The pure‑text “World‑Sim” corpus emphasizes indoor/architectural scenes; more diverse domains (e.g., underwater, crowd scenes) remain under‑explored.

Future directions include automated reward‑shaping via meta‑learning, integrating multi‑view 3‑D supervision, and scaling the approach to multi‑modal generation (audio‑synchronized video).

World‑R1 demonstrates that you don’t need to rebuild video diffusion models from scratch to get better 3‑D fidelity—just a smart reinforcement learning wrapper and the right textual world data can bridge the gap between flashy generative video and physically plausible virtual worlds. This opens the door for developers to embed more reliable, geometry‑aware video synthesis into their products with minimal engineering overhead.

Authors

• Weijie Wang
• Xiaoxuan He
• Youping Gu
• Yifan Yang
• Zeyu Zhang
• Yefei He
• Yanbo Ding
• Xirui Hu
• Donny Y. Chen
• Zhiyuan He
• Yuqing Yang
• Bohan Zhuang

Paper Information

• arXiv ID: 2604.24764v1
• Categories: cs.CV
• Published: April 27, 2026
• PDF: Download PDF