[Paper] How Low Can You Go? The Data-Light SE Challenge

Source: arXiv - 2512.13524v1

Overview

The paper challenges a common belief in software‑engineering research: that you need massive labeled datasets and heavyweight optimizers to get good results. By systematically testing dozens of SE problems—from configuration tuning to reinforcement‑learning‑based testing—the authors show that a handful of carefully chosen samples (often fewer than 50) can achieve ~90 % of the best published performance, using very simple algorithms.

Key Contributions

• Data‑light challenge definition – formalizes when a small number of labels is sufficient for SE tasks.
• Lightweight baselines – introduces and releases easy‑to‑implement methods (diversity sampling, a minimal Bayesian learner, random probing).
• Extensive empirical study – evaluates these baselines on a wide spectrum of SE problems (cloud optimization, project health prediction, financial risk, testing, etc.).
• Open‑science artifacts – provides all scripts, datasets, and a reproducible benchmark suite on GitHub.
• Insightful guidelines – identifies problem characteristics (e.g., smoothness of the objective, noise level) that predict when light methods will succeed.

Methodology

1. Problem Formalization – Each SE task is cast as a black‑box optimization or supervised‑learning problem where the goal is to find a configuration or predict an outcome with as few labeled instances as possible.
2. Labeling Model – The authors define a cost‑aware labeling budget and treat each “probe” (evaluation of a configuration or acquisition of a label) as a unit of expense.
3. Baseline Algorithms
   • Diversity Sampling – selects points that are maximally far apart in the feature space, ensuring coverage with few samples.
   • Minimal Bayesian Learner – a lightweight Gaussian‑process‑like model that updates with each new label but avoids expensive hyper‑parameter tuning.
   • Random Probes – a naïve baseline that serves as a sanity check.
4. Benchmark Suite – Over 30 publicly available SE datasets spanning multiple domains; each is run under identical labeling budgets (10, 20, 30, … 50 samples).
5. Comparison – Results are compared against state‑of‑the‑art optimizers (SMAC, TPE, DEHB, etc.) that typically consume thousands of evaluations.

Results & Findings

Key take‑aways

• Near‑optimal performance (≥ 90 % of the best) is routinely achieved with < 50 labels.
• Simple baselines match or outperform heavyweight methods on many noisy or low‑dimensional problems.
• Diminishing returns appear after ~30–40 samples; additional evaluations rarely improve the objective noticeably.

Practical Implications

• Faster prototyping – Teams can obtain actionable configuration recommendations in minutes rather than hours/days of compute.
• Cost savings – Reduces cloud‑compute spend for hyper‑parameter tuning or performance benchmarking, especially for small‑to‑medium projects.
• Embedded optimization – Light methods can run directly on edge devices or CI pipelines where CPU and memory are limited.
• Data‑efficient ML – Encourages developers to adopt active‑learning‑style sampling instead of brute‑force data collection, improving privacy and compliance (fewer user data points needed).
• Tooling impact – Existing SE toolchains (e.g., AutoML libraries, CI optimizers) could expose a “data‑light mode” that automatically switches to diversity sampling when the labeling budget is low.

Limitations & Future Work

• Problem scope – The study focuses on problems with relatively smooth search spaces; highly multimodal or adversarial settings may still need extensive sampling.
• Label noise – While the authors simulate measurement noise, real‑world noisy labels (e.g., flaky tests) could degrade the simple baselines more than sophisticated methods.
• Scalability to high dimensions – Diversity sampling can become less effective as dimensionality grows; future work should explore dimensionality reduction or adaptive sampling strategies.
• Integration studies – The paper calls for industry‑scale case studies to validate the data‑light approach in continuous‑delivery pipelines and large‑scale cloud environments.

Bottom line: For many everyday SE optimization tasks, “less is more.” A modest number of well‑chosen data points can give you most of the benefit of heavyweight tuning, freeing developers to iterate faster and spend less on compute.

Authors

• Kishan Kumar Ganguly
• Tim Menzies

Paper Information

• arXiv ID: 2512.13524v1
• Categories: cs.SE
• Published: December 15, 2025
• PDF: Download PDF