[Paper] SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior
Source: arXiv - 2606.11543v1
Overview
Agent Skills augment large language model (LLM) agents with procedural knowledge at inference time, but current benchmarks rarely distinguish what a Skill says from how it is organized. We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand, and compare it with a normalized flat baseline. We present SkillJuror, a framework for evaluating Skill writing paradigms through semantically controlled variants, matched multi-trial evaluations, and trajectory evidence while holding task knowledge fixed. In an 82-task SkillsBench study, Progressive Disclosure changes runtime behavior before aggregate outcomes: distinct Skill resources touched per trajectory rise from 1.18 to 3.85, and effective uptake events rise from 1.33 to 3.92. It also yields 17 additional verifier-passing trials out of 410 matched trials (+4.1%) over the normalized flat baseline. The benefit is task-dependent. Progressive Disclosure helps when supporting resources guide implementation, checking, or repair, but is weaker when success hinges on exact output conventions, numerical thresholds, or long artifact-generation pipelines. These results show that Skill organization is not mere presentation: it can change how agents search and apply procedural knowledge, while outcome gains depend on whether the exposed resources are actionable for the task. Code is available at https://github.com/zhiyuchen-ai/skill-juror.
Key Contributions
This paper presents research in the following areas:
- cs.AI
- cs.SE
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.AI.
Authors
- Zhiyu Chen
- Zihan Guo
- Bo Huang
- Bingwei Lu
- Jianghao Lin
- Yuanjian Zhou
- Weinan Zhang
Paper Information
- arXiv ID: 2606.11543v1
- Categories: cs.AI, cs.SE
- Published: June 10, 2026
- PDF: Download PDF