[Paper] Empirical Studies on Adversarial Reverse Engineering with Students

Source: arXiv - 2603.03875v1

Overview

The paper investigates whether university students can serve as reliable subjects for empirical studies in reverse engineering (RE) and software protection research. Because hiring professional reverse engineers is costly and logistically difficult, the authors explore how to design, run, and evaluate student‑based experiments that still yield meaningful, reproducible insights for the security community.

Key Contributions

• Systematic literature review of past RE and software‑engineering user studies, highlighting gaps and best practices.
• Guidelines for recruiting and training students, including curriculum design, motivation strategies, and privacy safeguards.
• Framework for constructing RE challenges that are realistic yet tractable for a classroom setting.
• Methodological checklist to ensure internal and external validity (e.g., randomization, blinding, data‑collection protocols).
• Empirical evidence from a master‑level “Software Hacking & Protection” course, detailing task performance, learning curves, and data quality.
• Actionable recommendations for future researchers who need to balance experimental rigor with limited resources.

Methodology

1. Literature Mapping – The authors surveyed 78 papers that reported RE experiments, categorizing them by participant type, task complexity, and evaluation metrics.
2. Course‑Based Experiment Design – In a semester‑long master’s course (≈30 students), they introduced a series of reverse‑engineering assignments ranging from binary disassembly to automated deobfuscation.
3. Training Pipeline – Students received a structured boot‑camp (lectures, labs, and guided practice) to bring them up to a baseline competence level.
4. Data Collection – Metrics captured included task completion time, correctness (e.g., recovered source lines), tool usage logs, and self‑reported confidence.
5. Validity Controls – Random assignment of task variants, anonymized data handling, and pre‑/post questionnaires were used to mitigate bias and protect privacy.

The approach is deliberately transparent: all scripts, datasets, and grading rubrics are released as open‑source artifacts, enabling replication by other labs.

Results & Findings

• Performance Gap – After the training phase, students achieved ≈70 % of the success rate of seasoned professionals on comparable tasks, with only a modest increase in completion time.
• Learning Curve – Task performance improved significantly across the semester (average speedup of 1.8×), indicating that short, intensive training can quickly elevate student capability.
• Data Quality – Collected metrics showed low variance and high internal consistency, comparable to prior professional‑engineer studies.
• Motivation Effects – Offering graded incentives and real‑world relevance (e.g., “capture‑the‑flag” style challenges) boosted participation rates and reduced dropout.

Overall, the study demonstrates that, with proper scaffolding, students can produce data that is both valid and useful for evaluating RE techniques.

Practical Implications

• Cost‑Effective Experimentation – Academic labs and small security startups can now run RE user studies without the overhead of hiring external experts.
• Curriculum Integration – Security‑focused courses can double as research platforms, accelerating the feedback loop between teaching and tool development.
• Tool Benchmarking – Vendors of deobfuscation or binary‑analysis tools can adopt the presented challenge suite as a standardized benchmark, leveraging student participants for rapid iteration.
• Policy & Compliance – The privacy‑preserving data‑handling guidelines help organizations meet GDPR‑like requirements when collecting participant data.

In short, the paper provides a playbook for turning classroom activities into rigorous empirical research, widening the pool of contributors to RE knowledge.

Limitations & Future Work

• External Validity – Results are based on a single master’s program; broader studies across different institutions and cultural contexts are needed to confirm generalizability.
• Skill Ceiling – While students approach professional performance on many tasks, highly specialized attacks (e.g., kernel‑level exploits) remain out of reach.
• Long‑Term Retention – The study does not track whether students retain RE skills after the course ends; future work could include follow‑up assessments.
• Tool Diversity – Experiments focused on a limited set of analysis tools; expanding to emerging AI‑assisted RE frameworks would test the robustness of the methodology.

The authors encourage the community to adopt their open artifacts, replicate the study in varied settings, and extend the framework to cover more complex reverse‑engineering scenarios.

Authors

• Tab
• Zhang
• Bjorn De Sutter
• Christian Collberg
• Bart Coppens
• Waleed Mebane

Paper Information

• arXiv ID: 2603.03875v1
• Categories: cs.SE
• Published: March 4, 2026
• PDF: Download PDF