[Paper] MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator

Source: arXiv - 2512.11782v1

Overview

The paper presents MatAnyone 2, a new framework that dramatically expands the scale and realism of video‑matting models. By introducing a learned Matting Quality Evaluator (MQE), the authors can both guide training in real time and automatically curate massive, high‑quality video‑matting data—resulting in a 28 K‑clip (2.4 M‑frame) dataset called VMReal. The approach pushes video‑matting performance to new state‑of‑the‑art levels on both synthetic and real‑world benchmarks.

Key Contributions

• Matting Quality Evaluator (MQE): A neural module that predicts pixel‑wise quality scores for alpha mattes without needing ground‑truth masks.
• Dual‑use of MQE:
  1. Online feedback during training to suppress low‑quality regions and provide richer supervision.
  2. Offline data curation that automatically selects and refines frames from existing video‑ and image‑matting models, enabling the creation of the large‑scale VMReal dataset.
• Reference‑frame training strategy: Incorporates long‑range temporal context beyond the usual short sliding window, improving robustness on long, appearance‑varying videos.
• VMReal dataset: 28 K diverse video clips (≈2.4 M frames) captured in the wild, filling a long‑standing gap in video‑matting resources.
• State‑of‑the‑art results: MatAnyone 2 outperforms prior methods across all standard metrics on both synthetic and real‑world test sets.

Methodology

1. Matting Quality Evaluator (MQE)

• Takes an RGB frame, a predicted alpha matte, and optionally a foreground/background estimate.
• Outputs a pixel‑wise quality map indicating confidence in the matte’s semantic consistency and boundary precision.
• Trained on a modest set of manually annotated mattes, learning to mimic human quality judgments.

2. Online Training Feedback

• During each training iteration, the MQE’s quality map weights the loss: high‑confidence regions contribute normally, while low‑confidence (error‑prone) pixels are down‑weighted.
• This dynamic supervision forces the matting network to focus on reliable patterns and reduces over‑fitting to noisy labels.

3. Offline Data Curation

• Run several strong video‑ and image‑matting models on raw video footage.
• Use MQE to score each resulting matte; frames with low scores are discarded or re‑processed.
• The remaining high‑quality mattes become the VMReal training set, dramatically expanding data volume without manual labeling.

4. Reference‑frame Training

• Instead of only using the immediate previous frame as a reference, the method samples long‑range frames (e.g., 5–10 seconds apart).
• This encourages the network to learn temporal consistency over larger appearance changes, which is crucial for real‑world videos where lighting, pose, and background can shift dramatically.

5. Network Architecture

• The core matting network follows an encoder‑decoder design with multi‑scale feature fusion, similar to prior video‑matting models, but now benefits from MQE‑guided loss and richer temporal cues.

Results & Findings

• Consistent gains across all metrics, especially on boundary‑sensitive measures (e.g., Trimap F‑score).
• Ablation studies show that removing MQE feedback degrades performance by ~7 % relative, while omitting reference‑frame training hurts long‑video stability.
• Qualitative examples demonstrate sharper hair strands, smoother semi‑transparent objects, and fewer flickering artifacts compared to previous methods.

Practical Implications

• Content creation pipelines: Studios can now generate high‑quality alpha mattes for VFX, AR/VR, and live‑streaming with far fewer manual rotoscoping hours.
• Real‑time applications: The MQE can be deployed as a lightweight quality monitor, flagging frames that need re‑processing in streaming or video‑conferencing tools.
• Dataset bootstrapping: Companies building proprietary matting models can use the MQE‑driven curation pipeline to quickly assemble domain‑specific datasets (e.g., sports broadcasts, e‑learning videos) without costly annotation.
• Improved downstream tasks: Better mattes boost performance of downstream segmentation, compositing, and background‑replacement APIs, leading to smoother user experiences in photo‑editing apps and virtual backgrounds.

Limitations & Future Work

• MQE training data: The evaluator still relies on a modest set of human‑rated mattes; its generalization to completely unseen domains (e.g., medical imaging) may be limited.
• Computational overhead: Running MQE alongside the matting network adds ~15 % extra inference time, which could be a bottleneck for ultra‑low‑latency scenarios.
• Dataset bias: VMReal, while large, is collected from publicly available video sources and may under‑represent niche lighting conditions or exotic materials.

Future research directions include self‑supervised MQE refinement, model compression for real‑time deployment, and expanding VMReal with active learning loops that continually ingest new video streams.

Authors

• Peiqing Yang
• Shangchen Zhou
• Kai Hao
• Qingyi Tao

Paper Information

• arXiv ID: 2512.11782v1
• Categories: cs.CV
• Published: December 12, 2025
• PDF: Download PDF