[Paper] Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology

Source: arXiv - 2602.16703v1

Overview

A pre‑registered, double‑blind randomized trial examined whether cutting‑edge large language models (LLMs) can boost the performance of novices tackling a multi‑step viral reverse‑genetics workflow in a real laboratory. Despite strong in‑silico results on biological benchmarks, the study found that LLM assistance did not dramatically raise the overall success rate, though modest gains were observed on several individual tasks.

Key Contributions

• First large‑scale RCT of LLMs in a wet‑lab setting – 153 participants, investigator‑blinded, with a control group using standard internet resources.
• Quantitative comparison of workflow completion – primary endpoint (full workflow success) showed no statistically significant difference (5.2 % vs. 6.6 %).
• Task‑level analysis – LLM users outperformed the control on four of five sub‑tasks, most notably cell‑culture (68.8 % vs. 55.3 %).
• Bayesian and ordinal regression modeling – suggests a ~1.4× increase in “typical” task success and a high posterior probability (81‑96 %) that LLMs improve intermediate step progression.
• Evidence of a gap between LLM performance on purely computational benchmarks and their practical utility in physical‑world bio‑experiments.

Methodology

1. Participants – 153 undergraduate‑level novices with minimal lab experience, randomly assigned to either an LLM‑assistance arm or a conventional Internet‑search arm.
2. Task suite – A five‑step reverse‑genetics pipeline (plasmid design, PCR, cloning, cell culture, viral rescue) that mirrors real‑world virology work.
3. Intervention – The LLM group used a state‑of‑the‑art conversational model (mid‑2025 release) for step‑by‑step guidance, while the control group consulted standard web resources (protocol sites, forums, etc.).
4. Blinding & pre‑registration – Researchers analyzing outcomes were blind to group allocation; the trial protocol was publicly registered before data collection.
5. Metrics – Primary outcome: full workflow completion. Secondary outcomes: success rates per task, number of intermediate steps completed, and time‑to‑completion.
6. Statistical analysis – Classical hypothesis testing (χ², Fisher’s exact) for primary/secondary endpoints, supplemented by Bayesian hierarchical models and ordinal regression to capture nuanced performance shifts.

Results & Findings

Takeaway: While LLMs did not make novices dramatically more likely to finish the entire pipeline, they provided a modest, statistically suggestive advantage on individual steps—especially the more hands‑on cell‑culture portion.

Practical Implications

• Tool selection for biotech startups – Teams can consider LLMs as a supplemental “virtual mentor” for routine protocol queries, but should not rely on them to replace hands‑on training or detailed SOPs.
• Safety and biosecurity policies – The modest performance lift suggests that LLMs alone are unlikely to enable large‑scale, unsupervised creation of viral constructs, easing some immediate dual‑use concerns.
• Developer focus – Building tighter integrations (e.g., LLMs that can query lab inventory systems, equipment APIs, or real‑time sensor data) may be needed to translate the observed step‑level gains into full‑workflow success.
• Education platforms – Incorporating LLM‑driven walkthroughs into virtual labs could improve learning outcomes for novice students, especially for tasks that are conceptually dense (e.g., cell culture).
• Benchmark design – The study underscores that benchmark suites limited to in‑silico tasks (sequence design, annotation) may overestimate real‑world impact; product roadmaps should include physical‑world validation loops.

Limitations & Future Work

• Participant expertise ceiling – Results reflect truly novice users; effects could differ for intermediate or expert technicians.
• LLM version – Only a single mid‑2025 model was tested; rapid model improvements may yield larger gains.
• Task scope – The reverse‑genetics workflow, while representative, is just one of many complex bio‑processes; generalization to other protocols (e.g., CRISPR editing, protein purification) remains open.
• Environmental variables – Lab equipment quality, instructor availability, and time pressure were not fully controlled, potentially diluting observable effects.
• Future directions – Planned studies will (1) evaluate multimodal models that can interpret images of gels or cell plates, (2) test LLMs in collaborative settings with human mentors, and (3) explore adaptive prompting strategies to reduce hallucinations in protocol advice.

Authors

• Shen Zhou Hong
• Alex Kleinman
• Alyssa Mathiowetz
• Adam Howes
• Julian Cohen
• Suveer Ganta
• Alex Letizia
• Dora Liao
• Deepika Pahari
• Xavier Roberts‑Gaal
• Luca Righetti
• Joe Torres

Paper Information

• arXiv ID: 2602.16703v1
• Categories: cs.CY, cs.AI
• Published: February 18, 2026
• PDF: Download PDF