[Paper] How to Trick Your AI TA: A Systematic Study of Academic Jailbreaking in LLM Code Evaluation

Source: arXiv - 2512.10415v1

Overview

The paper investigates a growing security risk: students can “jailbreak” large language models (LLMs) that are used as automatic graders for programming assignments. By crafting clever prompts, they can trick the AI into awarding higher scores than deserved. The authors conduct the first large‑scale, systematic study of these “academic jailbreaking” attacks and release a benchmark that will help the community build more robust grading systems.

Key Contributions

• Taxonomy of attacks – Adapted and extended 20+ known jailbreak techniques to the code‑evaluation setting, defining a new class called academic jailbreaking.
• Adversarial dataset – Released a “poisoned” corpus of 25 K student code submissions (real coursework, rubrics, and human‑graded references) engineered to fool LLM graders.
• Metrics suite – Introduced three quantitative measures: Jailbreak Success Rate (JSR), Score Inflation, and Harmfulness, to capture how badly an attack degrades grading quality.
• Empirical evaluation – Tested the attacks on six popular LLMs (e.g., GPT‑4, Claude, Llama 2). Persuasive and role‑play prompts achieved up to 97 % JSR, dramatically inflating scores.
• Open‑source benchmark – Provided code, prompts, and evaluation scripts so researchers and tool builders can stress‑test their grading pipelines.

Methodology

1. Prompt engineering – The authors took existing jailbreak recipes (e.g., “ignore previous instructions”, “pretend you are a helpful teacher”) and rewrote them to fit a typical academic grading workflow (e.g., “You are a professor grading this Python function”).
2. Dataset construction – Real student submissions from multiple universities were collected, each paired with a rubric and a human‑graded score. The team then applied the engineered prompts to generate adversarial versions of the same code, preserving the original logic but embedding the jailbreak cues.
3. Evaluation pipeline – Each LLM was fed the original and adversarial submissions along with the rubric. The model’s returned score was compared to the human baseline, and the three metrics (JSR, Score Inflation, Harmfulness) were computed.
4. Analysis – Results were broken down by attack family (persuasive, role‑play, instruction‑bypass, etc.) and by model size/architecture to understand which designs are most vulnerable.

Results & Findings

• High success rates: Persuasive and role‑play attacks consistently broke the grading logic, with JSR ranging from 70 % to 97 % across models.
• Score inflation: On average, adversarial prompts inflated grades by 12–18 % points, enough to turn a failing submission into a passing one.
• Model differences: Larger, instruction‑tuned models (e.g., GPT‑4) were not immune; they sometimes showed slightly lower JSR but still suffered significant inflation. Smaller open‑source models were even more susceptible.
• Harmfulness: Some attacks caused the grader to produce nonsensical feedback or reveal internal prompt‑engineering tricks, raising concerns about confidentiality and academic integrity.

Practical Implications

• Rethink AI‑based grading pipelines – Institutions should not rely on a single LLM call; instead, they need multi‑step verification (e.g., static analysis + LLM + human audit).
• Prompt hardening – Designing robust system prompts (e.g., “Never deviate from the rubric”, “Reject role‑play requests”) can reduce success rates, but the paper shows that even well‑crafted prompts can be bypassed.
• Monitoring & detection – The released adversarial dataset can be used to train detectors that flag suspiciously high scores or unusual language patterns in student submissions.
• Policy updates – Academic honesty policies may need to explicitly cover AI‑assisted cheating techniques, and educators should educate students about the ethical use of LLMs.
• Tool development – Developers building grading SaaS can integrate the benchmark to continuously test and harden their models before deployment, similar to security fuzzing for software.

Limitations & Future Work

• Scope of subjects – The study focuses on programming assignments; other domains (e.g., essays, design) may exhibit different vulnerabilities.
• Static dataset – While 25 K examples are extensive, attackers could evolve new prompts that bypass the current defenses, so continuous dataset updates are needed.
• Model coverage – Only six LLMs were evaluated; newer or more specialized models might behave differently.
• Defensive strategies – The paper primarily characterizes attacks; future work should explore systematic defenses (e.g., adversarial training, ensemble grading) and formal verification of grading prompts.

By exposing how easily LLM graders can be manipulated, this research gives developers, educators, and platform builders a concrete roadmap to safeguard automated code assessment from academic jailbreaks.

Authors

• Devanshu Sahoo
• Vasudev Majhi
• Arjun Neekhra
• Yash Sinha
• Murari Mandal
• Dhruv Kumar

Paper Information

• arXiv ID: 2512.10415v1
• Categories: cs.SE, cs.AI
• Published: December 11, 2025
• PDF: Download PDF