[Paper] On Assessing the Relevance of Code Reviews Authored by Generative Models

Source: arXiv - 2512.15466v1

Overview

The paper investigates how well large language models—specifically ChatGPT—can write code‑review comments. By introducing a new “multi‑subjective ranking” evaluation, the authors show that AI‑generated reviews can actually outperform the best human answers on a real‑world StackExchange dataset, highlighting both the promise and the perils of handing code‑review tasks over to generative AI.

Key Contributions

• Multi‑subjective ranking framework – a novel evaluation method that aggregates rankings from multiple human judges instead of relying on a single “ground truth” or vague usefulness scores.
• Large‑scale empirical study – 280 self‑contained code‑review requests from CodeReview StackExchange were paired with ChatGPT‑generated comments and the top human responses.
• Empirical finding – ChatGPT’s comments were consistently ranked higher than the accepted human answers, indicating that generative models can produce high‑quality review feedback.
• Risk awareness – the study surfaces the danger of blindly integrating AI reviews into development pipelines without proper validation.
• Open‑source dataset & evaluation scripts – the authors release the annotated dataset and ranking code to enable reproducibility and future benchmarking.

Methodology

1. Data collection – The authors scraped 280 code‑review questions from CodeReview StackExchange, each with at least one highly‑voted human answer.
2. AI generation – For every question, they prompted ChatGPT (GPT‑4) to produce a review comment, ensuring the prompt matched the style of the original request.
3. Human judging – Six independent developers (with varying experience levels) were recruited. Each judge received a randomised mix of three items per question: the ChatGPT comment, the accepted human answer, and the next‑best human answer.
4. Ranking task – Judges ordered the three comments from “most helpful” to “least helpful” based on clarity, correctness, actionable advice, and safety considerations.
5. Statistical aggregation – Rankings were converted to scores using the Bradley‑Terry model, allowing the authors to compute a pairwise win‑rate for AI vs. human comments and test significance with a Wilcoxon signed‑rank test.

Results & Findings

• Win‑rate: ChatGPT‑generated comments beat the accepted human answer in 62 % of the pairwise comparisons (p < 0.001).
• Quality dimensions: AI comments excelled in clarity and completeness but occasionally missed subtle security nuances that human experts caught.
• Inter‑rater agreement: Fleiss’ κ = 0.48, indicating moderate consensus among judges—enough to trust the ranking but also reflecting the inherent subjectivity of code‑review quality.
• Variability: While the average AI score was higher, a small tail (≈8 %) of AI comments were ranked worst, often due to hallucinated facts or outdated API usage.

Practical Implications

• Developer tooling: IDE plugins or CI bots could surface AI‑draft review comments as “first‑pass” feedback, letting human reviewers focus on higher‑level design or security concerns.
• Speed & cost: Teams can reduce the time spent on routine style or lint‑type feedback, potentially cutting review cycle times by 30‑40 % on large codebases.
• Training data loops: The multi‑subjective ranking method can be baked into continuous evaluation pipelines, automatically flagging AI‑generated comments that drift below a quality threshold.
• Safety nets: Because a minority of AI comments still contain factual errors, a lightweight verification step (e.g., static analysis or a quick human sanity check) should be mandatory before merging.
• Knowledge transfer: New hires can use AI‑generated reviews as learning material, seeing both the AI’s reasoning and the human reviewer’s final verdict.

Limitations & Future Work

• Domain scope: The dataset focuses on relatively small, self‑contained snippets; performance on large, multi‑module systems remains unknown.
• Judge diversity: All judges were software engineers; incorporating QA specialists, security auditors, or domain experts could reveal different quality trade‑offs.
• Model versioning: Only GPT‑4 was evaluated; future work should compare across model sizes and fine‑tuned variants to understand the impact of instruction tuning.
• Long‑term impact: The study does not measure how AI‑assisted reviews affect downstream bug rates or developer learning curves—longitudinal studies are needed.

Bottom line: The research shows that generative AI can already produce code‑review comments that rival—or even surpass—human experts in many cases, but responsible deployment demands robust evaluation, safety checks, and ongoing human oversight.

Authors

• Robert Heumüller
• Frank Ortmeier

Paper Information

• arXiv ID: 2512.15466v1
• Categories: cs.SE, cs.AI
• Published: December 17, 2025
• PDF: Download PDF