[Paper] ASMR-Bench: Auditing for Sabotage in ML Research

Source: arXiv - 2604.16286v1

Overview

The paper introduces ASMR‑Bench, a new benchmark that measures how well auditors—both humans and AI assistants—can spot deliberately sabotaged machine‑learning research code. As AI‑driven tools start writing and running experiments autonomously, the risk of hidden “backdoors” that skew results without obvious signs grows. ASMR‑Bench provides a concrete way to evaluate detection methods before such sabotage becomes a real production problem.

Key Contributions

• A sabotage‑focused benchmark: 9 real‑world ML research codebases, each paired with a subtly altered version that changes hyper‑parameters, data splits, or evaluation logic while keeping the paper’s narrative unchanged.
• Comprehensive evaluation protocol: Auditors are asked to (1) flag a codebase as sabotaged or clean, (2) locate the offending change, and (3) propose a fix.
• Baseline performance numbers: State‑of‑the‑art LLMs (e.g., Gemini 3.1 Pro) and LLM‑assisted human auditors achieve at most AUROC 0.77 and a top‑1 fix rate of 42 %.
• Red‑team analysis: LLMs can also generate sabotage scripts; while these are generally weaker than human‑crafted ones, they still manage to slip past auditors of comparable capability.
• Open‑source release: The full benchmark, evaluation scripts, and a leaderboard are made publicly available to spur research on automated auditing tools.

Methodology

1. Selection of codebases – The authors gathered nine publicly released ML research projects (e.g., image classification, reinforcement learning) that are representative of typical academic pipelines.
2. Sabotage injection – For each project, a human expert introduced a single, realistic flaw (e.g., swapping a learning‑rate schedule, truncating the training set, or mis‑computing a metric). The sabotage is designed to be plausible and to produce results that look “reasonable” but are actually incorrect.
3. Auditor tasks – Auditors receive the original repository and the sabotaged version (without any label). They must:
   • Detect whether sabotage is present (binary classification).
   • Identify the exact file/line responsible.
   • Repair the code to recover the original performance.
4. Evaluation metrics –
   • AUROC for detection (how well the auditor ranks clean vs. sabotaged).
   • Top‑1 fix rate (percentage of cases where the auditor’s suggested change restores the original result).
   • Time-to‑detect and explainability (qualitative analysis of audit reports).
5. LLM vs. human auditors – The study compares pure LLMs (prompted with the repository), LLM‑assisted humans (LLM suggestions plus human judgment), and a baseline of unaided human reviewers.

Results & Findings

• Detection is hard: Even the best model only correctly ranks sabotaged vs. clean code about three‑quarters of the time.
• Fixing is tougher: Less than half of the sabotages are fully corrected, indicating that spotting the bug does not automatically translate into a proper repair.
• LLM‑generated sabotage: When LLMs act as red‑teamers, the resulting sabotages are less subtle (AUROC ≈ 0.65 for same‑capability auditors) but still evade detection in a non‑trivial fraction of cases.
• Human intuition still matters: The modest edge of LLM‑assisted humans suggests that domain knowledge combined with AI suggestions yields the most reliable audits.

Practical Implications

• Automated research pipelines need watchdogs – Teams that let LLMs write experiments (e.g., hyper‑parameter sweeps, model‑selection scripts) should integrate an independent audit step, preferably using a different model or a human‑in‑the‑loop.
• CI/CD for ML research – The benchmark can be repurposed as a “security test” in continuous integration, flagging suspicious changes before they affect downstream products.
• Tooling opportunities – IDE plugins or GitHub actions that automatically run ASMR‑style checks could surface hidden metric mis‑computations or data‑leakage bugs early.
• Regulatory compliance – For industries where reproducibility is legally mandated (healthcare, finance), ASMR‑Bench provides a concrete metric to demonstrate due‑diligence in auditability.
• Red‑team training – Organizations can use the LLM‑generated sabotage scripts to train internal auditors, improving resilience against both human and AI adversaries.

Limitations & Future Work

• Scope of sabotage – The benchmark focuses on single‑point, subtle bugs. Real‑world sabotage could involve multi‑file, coordinated attacks or more sophisticated data poisoning, which are not covered.
• Domain coverage – All nine codebases are from standard supervised learning tasks; domains like NLP generation, graph learning, or large‑scale pre‑training remain untested.
• Auditor expertise variance – The study uses a limited pool of human auditors; broader crowdsourced evaluations could reveal different performance trends.
• Metric diversity – AUROC and top‑1 fix rate capture detection and repair but ignore partial fixes or the cost of false positives, which matter in production settings.
• Future directions – Extending ASMR‑Bench with multi‑step sabotage chains, integrating dynamic analysis (e.g., runtime profiling), and exploring ensemble auditor architectures are promising next steps.

ASMR‑Bench opens a much‑needed conversation about the hidden risks of AI‑generated research. By providing a concrete yardstick, it equips developers, ML engineers, and research managers with the tools to start building safer, more trustworthy autonomous experimentation pipelines.

Authors

• Eric Gan
• Aryan Bhatt
• Buck Shlegeris
• Julian Stastny
• Vivek Hebbar

Paper Information

• arXiv ID: 2604.16286v1
• Categories: cs.AI
• Published: April 17, 2026
• PDF: Download PDF