[Paper] Immersion in the GitHub Universe: Scaling Coding Agents to Mastery

Source: arXiv - 2602.09892v1

Overview

The paper introduces ScaleSWE, a fully automated, sandboxed multi‑agent pipeline that can harvest and curate real‑world software‑engineering (SWE) data at massive scale. By orchestrating three specialized agents—environment setup, test generation, and problem‑statement synthesis—the authors processed 6 million pull requests from 5,200 GitHub repositories, yielding a publicly released dataset of 100 k verified SWE instances, the largest of its kind. The authors also show how this data can be used to fine‑tune large language models (LLMs) into high‑performing coding assistants.

Key Contributions

• ScaleSWE pipeline: A reproducible, multi‑agent workflow that automatically builds end‑to‑end coding tasks (environment, tests, description) from raw pull‑request histories.
• ScaleSWE Data: 100 k high‑quality, verified software‑engineering instances covering a diverse set of languages, libraries, and project sizes—far surpassing existing benchmarks in both quantity and realism.
• Agent‑driven data generation: Demonstrates that three purpose‑built agents can reliably create correct test suites and coherent problem statements without human intervention.
• Model fine‑tuning: Distills 71 498 successful execution trajectories and fine‑tunes the Qwen‑30B‑BA3B‑Instruct model, producing the ScaleSWE Agent that resolves 64 % of tasks on the SWE‑Bench Verified benchmark (≈3× improvement over the base model).
• Open‑source release: Both the dataset and the pipeline code will be publicly available, enabling the community to replicate and extend the approach.

Methodology

1. Pull‑request mining – The system crawls GitHub to collect 6 M PRs from 5.2 k repositories, selecting those that modify code and include a merge commit.
2. Three‑agent orchestration
   • EnvAgent builds a reproducible sandbox (Docker/conda) that mirrors the repository’s dependencies and runtime.
   • TestAgent automatically generates unit tests for the changed code using a combination of static analysis, mutation testing, and LLM‑driven test synthesis.
   • PromptAgent crafts a concise problem description (the “coding task”) that a developer would see, based on commit messages, issue discussions, and code diffs.
3. Verification loop – The generated test suite is executed against the modified code; only instances that pass all tests are kept, guaranteeing functional correctness.
4. Trajectory extraction – For each verified instance, the system records the step‑by‑step interaction (prompt → model output → test execution) to create training trajectories.
5. Model fine‑tuning – The collected trajectories are used to fine‑tune Qwen‑30B‑BA3B‑Instruct, employing standard instruction‑tuning loss with reinforcement learning from human feedback (RLHF) to prioritize correct, concise solutions.

Results & Findings

• Dataset scale & diversity – 100 k verified instances spanning 30+ programming languages, with a median of 5 files per task and realistic dependency graphs.
• Baseline vs. fine‑tuned model – The base Qwen‑30B model solved ~22 % of SWE‑Bench Verified tasks; after fine‑tuning on ScaleSWE trajectories, the ScaleSWE Agent achieved a 64 % solve rate, a near‑tripling of performance.
• Ablation studies – Removing any of the three agents drops verification success by >30 %, confirming that each component is essential for high‑quality data.
• Human evaluation – Independent developers rated the generated problem statements as “clear and realistic” in 87 % of cases, indicating that the synthetic prompts are usable for training and benchmarking.

Practical Implications

• Better coding assistants – Developers can now integrate a model that has been trained on truly real‑world pull‑request scenarios, leading to suggestions that respect project conventions, dependency constraints, and test‑driven development practices.
• Accelerated tool building – Companies building automated code review, bug‑fix generation, or CI‑assistant tools can leverage the ScaleSWE dataset to bootstrap their models without the costly manual curation of training data.
• Continuous data pipeline – The multi‑agent workflow can be scheduled to run on fresh PR streams, enabling a living dataset that evolves with the open‑source ecosystem—useful for keeping LLMs up‑to‑date with emerging libraries and frameworks.
• Benchmarking & research – Researchers gain a large, verified benchmark for evaluating LLMs on realistic SWE tasks, moving beyond synthetic or toy examples that dominate current literature.

Limitations & Future Work

• Language bias – Although the dataset covers many languages, the majority of instances are still Python, JavaScript, and Java, reflecting GitHub’s language distribution; rarer languages remain under‑represented.
• Test quality ceiling – Automated test generation, while effective, may miss edge‑case bugs that human‑written tests would catch, potentially inflating the perceived solve rate.
• Compute cost – Running the sandboxed agents on millions of PRs requires substantial cloud resources, which may limit reproducibility for smaller labs.
• Future directions – The authors plan to (1) incorporate more sophisticated static analysis to improve test coverage, (2) extend the pipeline to handle multi‑module and micro‑service architectures, and (3) explore active learning loops where model failures trigger targeted data generation.

Authors

• Jiale Zhao
• Guoxin Chen
• Fanzhe Meng
• Minghao Li
• Jie Chen
• Hui Xu
• Yongshuai Sun
• Xin Zhao
• Ruihua Song
• Yuan Zhang
• Peng Wang
• Cheng Chen
• Jirong Wen
• Kai Jia

Paper Information

• arXiv ID: 2602.09892v1
• Categories: cs.SE
• Published: February 10, 2026
• PDF: Download PDF