[Paper] SparkVSR: Interactive Video Super-Resolution via Sparse Keyframe Propagation

Source: arXiv - 2603.16864v1

Overview

SparkVSR introduces a new way to do video super‑resolution (VSR) that puts the user back in control. Instead of treating the VSR model as a black box, the framework lets developers supply a handful of high‑resolution (HR) keyframes—either manually chosen or extracted automatically—and then intelligently propagates those details across the whole video while staying faithful to the original low‑resolution (LR) motion.

Key Contributions

• Interactive keyframe‑driven VSR – Users can guide the upscaling process with sparse HR keyframes, enabling correction of artifacts and artistic control.
• Two‑stage latent‑pixel training pipeline – Learns to fuse LR video latent features with encoded HR keyframe latents, achieving robust cross‑space propagation and fine‑grained detail refinement.
• Reference‑free guidance mechanism – Dynamically balances reliance on keyframes versus blind restoration, so the system remains stable even when keyframes are missing or imperfect.
• Flexible keyframe selection – Supports manual selection, automatic extraction of codec I‑frames, or random sampling without retraining.
• Generalizable framework – Demonstrated out‑of‑the‑box applicability to related video tasks such as old‑film restoration and style transfer.

Methodology

1. Keyframe Preparation – A developer runs any off‑the‑shelf image super‑resolution model (e.g., ESRGAN, SwinIR) on a sparse set of frames, producing HR keyframes.
2. Latent Encoding – Both the LR video and the HR keyframes are passed through separate encoders to obtain latent representations.
3. Two‑Stage Fusion
   • Stage 1: The LR latent stream is combined with the HR keyframe latents using a cross‑attention module that learns how to align motion while injecting high‑frequency details.
   • Stage 2: A pixel‑space refinement network cleans up any remaining artifacts, guided by a perceptual loss that encourages natural textures.
4. Reference‑Free Guidance – During inference, a gating network evaluates the confidence of each propagated keyframe region. When confidence is low (e.g., keyframe absent or mismatched), the model falls back to pure blind VSR, ensuring temporal consistency.
5. Training Objective – The loss combines reconstruction (L1/L2), perceptual (VGG‑based), and temporal consistency terms (optical‑flow warping loss) to teach the model to respect both motion and keyframe detail.

Results & Findings

• Quantitative Gains – SparkVSR outperforms strong baselines on three perceptual VSR benchmarks: +24.6 % on CLIP‑IQA, +21.8 % on DOVER, and +5.6 % on MUSIQ.
• Temporal Consistency – Visual inspection and flow‑based metrics show smoother frame‑to‑frame transitions, reducing flicker that often plagues VSR outputs.
• Robustness to Missing Keyframes – Even when only 5 % of frames are supplied as HR references, the model maintains high quality, thanks to the reference‑free gating.
• Cross‑Task Generalization – Without any task‑specific fine‑tuning, SparkVSR successfully restores degraded archival footage and applies artistic style transfer, confirming the versatility of the latent‑pixel fusion design.

Practical Implications

• Developer‑Friendly Pipelines – Teams can plug SparkVSR into existing media processing stacks, using their preferred ISR model for keyframe generation and letting SparkVSR handle the heavy lifting of temporal propagation.
• Interactive Editing Tools – Video editors can correct problematic frames on the fly (e.g., fixing a blurry face) by re‑rendering just those keyframes, saving compute compared to re‑processing the entire clip.
• Streaming & Bandwidth Optimization – Content providers could transmit a low‑resolution stream plus a few high‑resolution keyframes (or I‑frames) and let the client device upscale the rest, reducing bandwidth while preserving visual fidelity.
• Legacy Media Restoration – Archivists can upscale old films by manually enhancing a few representative frames; SparkVSR will propagate those improvements throughout the footage, accelerating restoration workflows.

Limitations & Future Work

• Keyframe Dependency – While the system degrades gracefully, the best results still rely on well‑chosen HR keyframes; poor or misaligned keyframes can introduce artifacts.
• Computational Overhead – The two‑stage latent‑pixel pipeline adds latency compared to end‑to‑end black‑box VSR models, which may be a concern for real‑time streaming scenarios.
• Generalization to Extreme Motions – Very fast or non‑linear motion can challenge the cross‑attention alignment, suggesting a need for more robust motion modeling.

Future research directions include adaptive keyframe selection strategies (e.g., learning which frames would yield maximal quality gain), lightweight encoder designs for on‑device inference, and tighter integration with video codecs to exploit existing I‑frame structures.

Authors

• Jiongze Yu
• Xiangbo Gao
• Pooja Verlani
• Akshay Gadde
• Yilin Wang
• Balu Adsumilli
• Zhengzhong Tu

Paper Information

• arXiv ID: 2603.16864v1
• Categories: cs.CV, cs.AI
• Published: March 17, 2026
• PDF: Download PDF