[Paper] Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025
Published: (December 15, 2025 at 09:15 PM EST)
4 min read
Source: arXiv
Source: arXiv - 2512.14012v1
Overview
The paper Professional Software Developers Don’t Vibe, They Control: AI Agent Use for Coding in 2025 investigates how seasoned developers actually work with AI coding assistants (e.g., Copilot, GPT‑4‑based agents) on real projects. By combining on‑site observations with a large‑scale survey, the authors reveal that developers treat agents as powerful tools—not as autonomous partners—and deliberately steer their behavior to preserve code quality, maintainability, and other core software attributes.
Key Contributions
- Empirical portrait of AI‑agent usage: 13 in‑depth field observations and 99 survey responses from professional developers provide a rich, mixed‑methods view of current practices.
- Control‑oriented usage model: Introduces the “Control‑First” paradigm, where developers actively shape agent output rather than passively accepting suggestions.
- Task suitability taxonomy: Classifies coding tasks (e.g., boilerplate generation, API lookup, test scaffolding) that developers find most beneficial for agents versus those they keep fully manual.
- Design guidelines for agent interfaces: Derives concrete UI/UX recommendations (e.g., explicit “confidence” signals, fine‑grained prompt controls) to help agents fit into existing development workflows.
- Positive sentiment despite limitations: Shows that experienced developers are overall optimistic about agents, provided they can compensate for known shortcomings (e.g., hallucinations, style drift).
Methodology
- Field Observations (N=13) – Researchers shadowed developers in three tech companies over several weeks, recording how agents were invoked, edited, and integrated into daily coding tasks.
- Qualitative Survey (N=99) – An online questionnaire collected developers’ motivations, perceived benefits, frustrations, and strategies for controlling agents. Open‑ended responses were coded using thematic analysis.
- Triangulation – Findings from the observations were cross‑validated against survey themes to ensure consistency and to surface nuanced patterns (e.g., differences between junior vs. senior engineers).
The mixed‑methods approach keeps the technical depth while remaining understandable: think of it as “watching developers code with AI and then asking them why they did what they did.”
Results & Findings
| Finding | What it means |
|---|---|
| Developers view agents as “productivity boosters” but retain ultimate decision‑making. | Agents are used for repetitive or low‑risk code (e.g., getters, test stubs), while design‑level decisions stay human‑driven. |
| Control strategies are explicit: prompting with constraints, reviewing generated snippets line‑by‑line, and using version‑control diffs to reject unwanted changes. | Shows a mature workflow where the developer’s expertise guides the AI, reducing the risk of low‑quality output. |
| Task suitability: agents excel at search‑and‑replace, API usage examples, and boilerplate generation; they struggle with architectural design, security‑critical code, and domain‑specific business logic. | Provides a practical checklist for teams to decide where to invest AI assistance. |
| Positive sentiment: 78 % of surveyed developers expressed confidence that agents will become a standard part of the toolbox, citing “time saved” and “idea sparking” as top benefits. | Indicates market readiness for more sophisticated, controllable agents. |
| Pain points: hallucinated code, inconsistent style, and lack of transparent confidence scores were the most cited frustrations. | Highlights concrete gaps that need to be addressed in next‑gen tools. |
Practical Implications
- Tooling teams can prioritize controllability: Adding UI knobs for “temperature”, “max tokens”, or “strict mode” lets developers fine‑tune the agent’s behavior without leaving the IDE.
- Integrate agent output into CI pipelines: Automated linting and static analysis can catch style drift or security issues before a developer reviews the code, reinforcing the “control‑first” mindset.
- Adopt a task‑allocation matrix: Teams can map out which parts of the development lifecycle are AI‑friendly (e.g., test scaffolding) and which remain human‑only (e.g., core algorithm design).
- Training and onboarding: Since experienced developers already use control strategies, junior engineers should be taught these practices early to avoid over‑reliance on AI suggestions.
- Product roadmaps: Vendors should expose confidence scores, provenance metadata (which model generated which snippet), and easy “undo” mechanisms to align with developers’ need for agency.
Limitations & Future Work
- Sample bias: The field study focused on a handful of companies that already adopt AI agents; results may differ in more conservative or legacy‑heavy environments.
- Self‑reporting: Survey responses rely on participants’ perception of their own behavior, which can be subject to hindsight bias.
- Rapid tech evolution: The study captures a snapshot of 2024‑25 tools; newer models could shift task suitability dramatically.
Future research directions suggested by the authors include longitudinal studies to see how control strategies evolve as agents become more capable, and experimental work on UI designs that make agent confidence and intent more transparent to developers.
Authors
- Ruanqianqian Huang
- Avery Reyna
- Sorin Lerner
- Haijun Xia
- Brian Hempel
Paper Information
- arXiv ID: 2512.14012v1
- Categories: cs.SE, cs.AI, cs.HC
- Published: December 16, 2025
- PDF: Download PDF