[Paper] V-RGBX: Video Editing with Accurate Controls over Intrinsic Properties

Source: arXiv - 2512.11799v1

Overview

The paper introduces V‑RGBX, the first end‑to‑end system that can both understand and edit intrinsic properties of a video—such as albedo, surface normals, material parameters, and lighting—while keeping the output photorealistic and temporally stable. By coupling inverse rendering with a generative video model, V‑RGBX lets creators edit a few keyframes (e.g., change a car’s paint or relight a room) and have those changes automatically propagate across the entire clip in a physically plausible way.

Key Contributions

• Unified Intrinsic‑Aware Pipeline – Combines video inverse rendering, intrinsic‑conditioned synthesis, and keyframe‑based editing into a single trainable framework.
• Interleaved Conditioning Mechanism – A novel way to inject intrinsic maps (albedo, normals, material, irradiance) into a video diffusion model, enabling fine‑grained, physically grounded control.
• Temporal Consistency Guarantees – Architecture and loss design enforce frame‑to‑frame coherence, avoiding flicker that plagues many video‑to‑video models.
• Keyframe Editing Interface – Users edit any intrinsic channel on a sparse set of frames; the system automatically propagates edits throughout the video.
• State‑of‑the‑Art Results – Demonstrates superior visual quality and edit fidelity over prior video‑editing and intrinsic‑decomposition methods on several benchmarks.

Methodology

1. Video Inverse Rendering – A backbone encoder processes the input video and predicts per‑frame intrinsic maps:

   • Albedo (diffuse color)
   • Normal (surface orientation)
   • Material (specular/roughness)
   • Irradiance (lighting)

   These maps are learned jointly with a reconstruction loss that encourages the rendered image (using a simple differentiable renderer) to match the original frames.

2. Intrinsic‑Conditioned Video Synthesis – A video diffusion model (a 3‑D UNet operating on space‑time tensors) takes the intrinsic maps as conditioning inputs. The “interleaved conditioning” alternates between injecting low‑level (pixel‑wise) and high‑level (global) intrinsic features at multiple diffusion steps, giving the generator fine control over appearance while preserving motion cues.

3. Keyframe Editing Loop – Users modify any intrinsic map on a small set of keyframes (e.g., paint a car red, brighten a window). The edited maps replace the originals for those frames, and the diffusion model re‑generates the video conditioned on the mixed intrinsic sequence. A temporal propagation loss ensures the edited properties flow smoothly to neighboring frames.

4. Training Objectives – The system optimizes a combination of:

   • Reconstruction loss for inverse rendering
   • Diffusion denoising loss for synthesis
   • Temporal consistency loss (optical‑flow‑guided)
   • Intrinsic regularization (smoothness, physical plausibility)

Results & Findings

• Photorealism & Consistency – V‑RGBX achieves higher PSNR/SSIM and lower temporal warping error than baselines (e.g., video‑to‑video GANs, frame‑wise diffusion).
• Edit Fidelity – Quantitative metrics (e.g., L2 error on edited albedo) show that changes made on keyframes are accurately reproduced across the whole clip, even under complex motion.
• User Study – Participants preferred V‑RGBX outputs over competing tools for tasks like “change the color of a moving car” and “relight an indoor scene,” citing realism and lack of flicker.
• Speed – While diffusion models are compute‑heavy, the authors report ~2‑3 × faster inference than naïve per‑frame diffusion because the intrinsic maps are reused across time.

Practical Implications

• Content Creation Pipelines – V‑RGBX can be integrated into VFX or game‑asset pipelines to quickly prototype lighting or material changes without re‑rendering the whole scene.
• AR/VR Real‑Time Editing – The intrinsic maps can be stored once and reused for on‑device relighting or recoloring, enabling interactive experiences with minimal bandwidth.
• Automated Post‑Production – Studios could automate tedious tasks like color grading or object‑level retouching across long takes, freeing artists to focus on creative decisions.
• Data Augmentation – Synthetic video datasets with controllable intrinsic variations (e.g., different weather or material conditions) can be generated for training robust perception models.

Limitations & Future Work

• Compute Requirements – The diffusion backbone still needs high‑end GPUs for reasonable latency; real‑time editing remains out of reach.
• Intrinsic Ambiguities – In highly specular or translucent scenes, the inverse rendering step can produce ambiguous albedo/normal splits, limiting edit accuracy.
• Limited Modalities – Current implementation handles only four intrinsic channels; extending to subsurface scattering or volumetric lighting would broaden applicability.
• User Interface – The paper demonstrates keyframe editing via scripts; a polished UI for non‑technical artists is still an open engineering challenge.

Overall, V‑RGBX marks a significant step toward physically grounded, user‑friendly video editing, opening new possibilities for developers building next‑generation visual content tools.

Authors

• Ye Fang
• Tong Wu
• Valentin Deschaintre
• Duygu Ceylan
• Iliyan Georgiev
• Chun-Hao Paul Huang
• Yiwei Hu
• Xuelin Chen
• Tuanfeng Yang Wang

Paper Information

• arXiv ID: 2512.11799v1
• Categories: cs.CV
• Published: December 12, 2025
• PDF: Download PDF