[Paper] GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?

Source: arXiv - 2602.06013v1

Overview

The paper GenArena tackles a pressing problem in computer‑vision research: how to evaluate the output of modern visual generation models (e.g., text‑to‑image, image‑inpainting, video synthesis) in a way that truly reflects human judgment. The authors show that the widely‑used “absolute pointwise scoring” approach—where a model assigns a single quality score to each generated image—fails to be stable or human‑aligned. By switching to a pairwise comparison framework, they achieve dramatically higher correlation with human rankings and even let open‑source models beat proprietary giants on benchmark leaderboards.

Key Contributions

• Systematic critique of pointwise scoring – Empirical evidence that absolute scores are noisy, inconsistent across runs, and poorly correlated with human perception.
• GenArena framework – A unified, task‑agnostic evaluation pipeline that uses pairwise comparisons (A vs. B) instead of single‑image scores.
• Open‑source superiority – Demonstrates that, under the pairwise protocol, freely available models can surpass top‑tier commercial systems on several visual generation benchmarks.
• Large‑scale validation – Achieves a Spearman correlation of 0.86 with the human‑curated LMArena leaderboard—over 20 % absolute improvement compared to the 0.36 correlation of pointwise methods.
• Comprehensive benchmark suite – Applies GenArena to a wide range of tasks (text‑to‑image, image editing, video generation, etc.), providing the community with a ready‑to‑use, automated evaluation standard.

Methodology

1. Problem formulation – Treat evaluation as a ranking problem: given two generated outputs for the same prompt, decide which one looks more realistic or better satisfies the prompt.
2. Pairwise judgment model – Fine‑tune off‑the‑shelf Vision‑Language Models (VLMs) to predict a binary preference (A > B or B > A). The model receives the prompt, the two images, and outputs a confidence score for each direction.
3. Aggregation to a global ranking – Feed pairwise outcomes into a Bradley‑Terry or Mallows model to infer a consistent overall score for each system across many prompts, eliminating stochastic variance that plagues pointwise scores.
4. Human‑ground‑truth collection – A subset of prompts is evaluated by crowdsourced workers to build a gold‑standard ranking (the LMArena leaderboard), serving as the reference for correlation analysis.
5. Benchmarking pipeline – Run the same pairwise evaluator on dozens of state‑of‑the‑art generators, producing a reproducible leaderboard.

The approach is deliberately lightweight: it reuses existing VLMs (e.g., CLIP, BLIP) without needing expensive human annotation for every new model or task.

Results & Findings

• Stability: Re‑running the pairwise evaluator yields < 1 % variance in rankings, whereas pointwise scores swing by > 10 % across seeds.
• Open‑source advantage: Models such as Stable Diffusion 2.1 and DeepFloyd‑IF, when judged with GenArena, outrank commercial APIs (e.g., DALL·E 3) on the same prompts.
• Task generality: The same pairwise evaluator works across image generation, editing, and short video synthesis without task‑specific tuning.

Practical Implications

• More reliable model selection – Developers can trust the GenArena leaderboard to pick the best generator for a product (e.g., UI mock‑up tools, game asset pipelines) without costly human studies.
• Accelerated R&D cycles – Because the evaluation is fully automated, teams can iterate on model architecture or prompt engineering and get immediate, human‑aligned feedback.
• Open‑source democratization – Companies can achieve “state‑of‑the‑art” visual generation quality using free models, reducing reliance on expensive proprietary APIs.
• Standardization for competitions – GenArena offers a reproducible, cross‑task metric that could replace the fragmented pointwise scores currently used in many vision‑generation challenges.
• Integration with CI/CD – The pairwise evaluator can be wrapped as a test step in continuous integration pipelines, flagging regressions in visual fidelity early.

Limitations & Future Work

• Dependence on VLM quality – Pairwise judgments inherit biases from the underlying Vision‑Language Model; misinterpretations of visual concepts can skew rankings.
• Scalability of pairwise comparisons – Efficient sampling (e.g., tournament brackets) keeps the number of comparisons manageable, but extremely large model suites may still incur non‑trivial compute costs.
• Prompt diversity – The benchmark focuses on English prompts; extending to multilingual or highly domain‑specific prompts may require additional fine‑tuning.
• Human alignment beyond aesthetics – Current evaluations emphasize visual realism and prompt adherence; future work could incorporate higher‑level criteria such as creativity or ethical considerations.

The authors suggest exploring hybrid metrics that combine pairwise judgments with lightweight pointwise cues, and investigating how to adapt GenArena for emerging modalities like 3‑D asset generation and interactive visual agents.

Authors

• Ruihang Li
• Leigang Qu
• Jingxu Zhang
• Dongnan Gui
• Mengde Xu
• Xiaosong Zhang
• Han Hu
• Wenjie Wang
• Jiaqi Wang

Paper Information

• arXiv ID: 2602.06013v1
• Categories: cs.CV, cs.AI
• Published: February 5, 2026
• PDF: Download PDF