[Paper] Tuning-free Visual Effect Transfer across Videos

Source: arXiv - 2601.07833v1

Overview

The paper introduces RefVFX, a feed‑forward framework that can copy complex, time‑varying visual effects—from flickering lights to character transformations—seen in a reference video and apply them to a completely different target video or even a single image. By removing the need for per‑effect fine‑tuning or cumbersome textual prompts, RefVFX opens the door to “plug‑and‑play” video editing that works on any content while preserving the original motion and appearance.

Key Contributions

• Reference‑conditioned effect transfer: A model that directly consumes a reference video and a target, producing a temporally coherent output without any additional training per effect.
• Large‑scale synthetic dataset: A pipeline that automatically generates millions of triplets (reference + input → output) covering a wide variety of repeatable effects, including both video‑to‑video and image‑to‑video scenarios.
• Integration with modern text‑to‑video backbones: RefVFX builds on state‑of‑the‑art diffusion models, leveraging their generative power while adding a lightweight reference encoder.
• Empirical superiority: Quantitative metrics (FID, CLIP‑Video similarity) and human preference studies show RefVFX beats prompt‑only baselines and matches or exceeds specialist tools that require manual tuning.

Methodology

1. Data Generation

   • Effect‑preserving pipelines: The authors script deterministic visual transformations (e.g., color‑grade cycles, particle systems, facial morphs) and apply them to source videos, guaranteeing that the underlying motion stays intact.
   • LoRA‑based adapters: For more artistic effects, low‑rank adapters are trained on image‑to‑video pairs and then used to synthesize paired videos.
   • Triplet construction: Each sample consists of (a) a reference effect video (the “style”), (b) an input video or image (the content to be edited), and (c) the ground‑truth output where the effect has been transferred.

2. Model Architecture

   • Backbone: A pretrained text‑to‑video diffusion model (e.g., Stable Diffusion Video) provides the core generative capacity.
   • Reference Encoder: A 3‑D CNN extracts spatio‑temporal embeddings from the reference video. These embeddings are injected into the diffusion UNet via cross‑attention layers, allowing the model to condition on the effect dynamics.
   • Training: The system is trained end‑to‑end on the synthetic triplets using a standard diffusion loss, with no per‑effect fine‑tuning required at inference time.

3. Inference

   • Users supply a reference clip and a target (video or image). The model runs a single forward pass, producing an edited video that mirrors the reference’s temporal pattern while preserving the target’s content and motion.

Results & Findings

• Visual quality: RefVFX consistently generates sharp, artifact‑free frames that follow the reference’s timing (e.g., pulsating light, rhythmic color shifts).
• Temporal coherence: Metrics that penalize flicker (temporal SSIM, warping error) are significantly lower than those of prompt‑only baselines, indicating smoother motion.
• Generalization: The model successfully transfers unseen effect categories (e.g., a new particle system) despite never having seen that exact style during training.
• Human study: In a blind pairwise comparison, participants preferred RefVFX outputs over the best prompt‑driven alternative 78% of the time.

Practical Implications

• Rapid prototyping for VFX artists: Instead of hand‑crafting keyframes or writing complex shader scripts, artists can record a short reference clip of the desired effect and instantly apply it to any scene.
• Content creation at scale: Social media creators, game developers, and advertisers can automate repetitive visual motifs (e.g., brand‑specific lighting cycles) across large libraries of footage.
• Low‑cost post‑production: Small studios lacking dedicated VFX pipelines can achieve professional‑grade temporal effects with a single model inference, reducing both time and budget.
• Integration hooks: Because RefVFX runs in a feed‑forward manner on GPU, it can be wrapped as a plugin for popular video editors (Premiere, DaVinci Resolve) or exposed via an API for cloud‑based video processing services.

Limitations & Future Work

• Synthetic bias: The training data, while massive, is generated by scripted effects; extremely organic or chaotic real‑world phenomena (e.g., fire, water) may not transfer perfectly.
• Resolution & length: Current experiments focus on 256‑512 px clips up to a few seconds; scaling to 4K, long‑form content will require memory‑efficient architectures or chunked processing.
• Effect granularity: The model assumes a single dominant effect per reference; compositing multiple overlapping effects remains an open challenge.
• Future directions: The authors suggest expanding the dataset with captured real‑world effect videos, exploring hierarchical conditioning for multi‑effect blending, and optimizing for real‑time inference on edge devices.

Authors

• Maxwell Jones
• Rameen Abdal
• Or Patashnik
• Ruslan Salakhutdinov
• Sergey Tulyakov
• Jun-Yan Zhu
• Kuan-Chieh Jackson Wang

Paper Information

• arXiv ID: 2601.07833v1
• Categories: cs.CV
• Published: January 12, 2026
• PDF: Download PDF