[Paper] Plenoptic Video Generation

Source: arXiv - 2601.05239v1

Overview

PlenopticDreamer tackles a long‑standing problem in generative video re‑rendering: keeping multiple camera views consistent over time. While existing methods can synthesize high‑quality video from a single viewpoint, they often produce jittery or mismatched results when the camera moves or when several viewpoints are required. This paper presents a new framework that synchronizes the “hallucinated” content across space and time, enabling reliable multi‑view video generation for applications ranging from virtual production to robotic tele‑operation.

Key Contributions

• PlenopticDreamer framework – a multi‑in‑single‑out video‑conditioned generative model that enforces spatio‑temporal coherence across arbitrary camera trajectories.
• Camera‑guided video retrieval – an adaptive mechanism that selects the most relevant previously generated frames as conditioning inputs, ensuring that new frames stay aligned with past visual context.
• Progressive context scaling & self‑conditioning – training tricks that gradually increase the temporal window and feed the model its own past outputs, dramatically reducing error accumulation in long sequences.
• Long‑video conditioning – a strategy that allows the model to generate extended videos (hundreds of frames) without sacrificing quality or view consistency.
• State‑of‑the‑art results – on the Basic and Agibot benchmarks, PlenopticDreamer outperforms prior re‑rendering systems in view synchronization, visual fidelity, and camera control flexibility.

Methodology

1. Autoregressive video‑conditioned generation – The model receives a short clip (e.g., 4–8 frames) and a target camera pose, then predicts the next frame. This process repeats, feeding each newly generated frame back into the model.
2. Camera‑guided retrieval – Before generating a frame, the system queries a memory bank of previously generated frames, selecting those whose camera parameters are closest to the current target pose. These retrieved frames are concatenated with the current conditioning clip, giving the network a richer spatial context.
3. Progressive context scaling – Training starts with a small temporal window (few frames) and gradually expands to longer windows, helping the network learn short‑term dynamics before tackling long‑range dependencies.
4. Self‑conditioning – The model is also trained to predict the next frame when given its own past predictions as input, which improves robustness when inference inevitably introduces small errors.
5. Long‑video conditioning – For very long sequences, a hierarchical conditioning scheme splits the video into overlapping segments, each conditioned on the previous segment’s summary representation, preserving global coherence.

All components are built on top of a diffusion‑based generative backbone, but the innovations lie in how temporal and camera information are orchestrated rather than in the underlying image synthesis engine.

Results & Findings

• View synchronization – PlenopticDreamer reduces multi‑view drift by up to 45 % compared with ReCamMaster on the Basic benchmark, measured via pixel‑wise reprojection error.
• Visual quality – Frechet Video Distance (FVD) improves from 210 (baseline) to 132, indicating sharper, more realistic frames.
• Camera control accuracy – The generated videos follow prescribed camera trajectories with sub‑pixel error, enabling precise third‑person‑to‑first‑person transformations.
• Diverse transformations – Demonstrated on robotic manipulation tasks, the model can seamlessly switch from a head‑mounted view to a gripper‑mounted view while preserving object textures and motion dynamics.
• Scalability – Successful generation of videos up to 300 frames (≈10 s at 30 fps) without noticeable degradation, a regime where prior methods typically collapse.

Practical Implications

• Virtual production & VFX – Filmmakers can now generate consistent multi‑camera shots from a single captured sequence, reducing the need for expensive multi‑camera rigs.
• Robotics tele‑operation – Operators can request arbitrary viewpoints (e.g., from a robot’s wrist) on‑the‑fly, with the system delivering temporally coherent visual feedback, improving situational awareness.
• AR/VR content creation – Game developers and immersive experience designers can synthesize panoramic or stereoscopic video content that stays stable as users move their heads.
• Data augmentation – Training perception models for autonomous systems often requires multi‑view video; PlenopticDreamer can generate realistic, synchronized augmentations, potentially boosting model robustness.

Limitations & Future Work

• Computational cost – The autoregressive pipeline and retrieval step are memory‑intensive, making real‑time generation on edge devices challenging.
• Dependence on accurate camera metadata – Errors in pose estimation can propagate, leading to misaligned views; integrating pose refinement could mitigate this.
• Generalization to highly dynamic scenes – Extremely fast motions or large occlusions still cause occasional flickering; future work may explore hybrid physics‑based priors.
• Extending beyond diffusion backbones – Investigating more efficient architectures (e.g., transformer‑based video generators) could further accelerate inference.

Overall, PlenopticDreamer marks a significant step toward practical, multi‑view generative video systems, opening new doors for developers building immersive, camera‑controlled experiences.

Authors

• Xiao Fu
• Shitao Tang
• Min Shi
• Xian Liu
• Jinwei Gu
• Ming-Yu Liu
• Dahua Lin
• Chen-Hsuan Lin

Paper Information

• arXiv ID: 2601.05239v1
• Categories: cs.CV
• Published: January 8, 2026
• PDF: Download PDF