[Paper] CIFE: Code Instruction-Following Evaluation

Source: arXiv - 2512.17387v1

Overview

The paper “CIFE: Code Instruction‑Following Evaluation” tackles a gap in current code‑generation benchmarks: they mostly measure whether generated code passes test cases, but they ignore whether the code respects the developer’s explicit constraints (style, security, performance, etc.). By introducing a 1,000‑task benchmark with richly annotated constraints and a new composite metric (the C2A Score), the authors provide a more realistic yardstick for trustworthy code generation.

Key Contributions

• A large‑scale, constraint‑rich benchmark: 1,000 Python programming tasks, each paired with ~7 developer‑specified constraints covering 13 categories (e.g., naming conventions, memory limits, security checks).
• Human‑LLM pipeline for constraint curation: A four‑stage process that guarantees constraints are atomic, relevant, and objectively verifiable.
• Dual adherence metrics:
  • Partial adherence – counts constraints that are at least hinted at in the output.
  • Strict adherence – requires full, verifiable satisfaction of each constraint.
• C2A Score: A composite measure that jointly evaluates functional correctness (via test cases) and constraint compliance, enabling apples‑to‑apples comparison across models.
• Comprehensive evaluation: 14 open‑ and closed‑source LLMs (including top‑tier models) are benchmarked, revealing a pronounced gap between partial and strict adherence.

Methodology

1. Task selection – 1,000 diverse Python problems were drawn from existing coding datasets (e.g., HumanEval, MBPP) to ensure a mix of algorithmic difficulty and real‑world relevance.
2. Constraint generation – For each task, a human‑LLM workflow was used:
   • Human seed: developers write a brief natural‑language requirement list.
   • LLM expansion: a language model proposes additional constraints.
   • Human vetting: experts prune, refine, and ensure atomicity.
   • LLM verification: a second model checks that each constraint is objectively testable.
3. Model inference – All 14 models generate code given the task description and the full constraint list.
4. Evaluation pipeline –
   • Correctness: standard unit‑test execution.
   • Constraint checking: automated static analysis, runtime guards, and security scanners (e.g., Bandit) are run against the generated code.
   • Scoring: partial vs. strict adherence are computed, then combined with correctness into the C2A Score (weighted harmonic mean).

Results & Findings

• Partial vs. strict gap: Even the best models satisfy >90 % of constraints in a partial sense, but strict compliance stalls around 40–66 %.
• Constraint category impact: Security‑related constraints (e.g., “no use of eval”) and performance limits (e.g., “≤ O(n log n)”) are the hardest to meet strictly.
• C2A correlation: Models that excel in correctness do not automatically excel in constraint adherence; the composite score surfaces this divergence.

Practical Implications

• Tooling for CI/CD: The benchmark’s constraint‑checking suite can be integrated into continuous‑integration pipelines to automatically flag generated code that violates style, security, or performance policies.
• Prompt engineering: Developers should explicitly list constraints in prompts; however, the study shows that merely stating them is insufficient—LLMs need better instruction‑following capabilities.
• Model selection: When choosing a code‑generation model for production, teams should look beyond test‑case pass rates and consider C2A‑type metrics to gauge trustworthiness.
• Security audits: The findings underscore that even state‑of‑the‑art models can emit insecure code; coupling LLMs with static analysis tools remains essential.
• Custom constraint libraries: Companies can extend the 13‑category taxonomy to encode internal coding standards, then reuse the same evaluation pipeline to benchmark proprietary LLMs.

Limitations & Future Work

• Language scope: The benchmark is limited to Python; extending to other ecosystems (JavaScript, Go, Rust) is needed to generalize conclusions.
• Constraint granularity: While the pipeline strives for atomic constraints, some nuanced requirements (e.g., “maintain readability for junior developers”) remain hard to formalize and automatically verify.
• Model prompting bias: All models received the same raw constraint list; future work could explore richer prompting strategies (e.g., constraint hierarchy, examples) to see if adherence improves.
• Human‑in‑the‑loop evaluation: Strict adherence is currently measured by automated tools; a manual audit could surface false negatives/positives, especially for security‑related constraints.

Bottom line: CIFE shines a light on the next frontier for code‑generation AI—moving from “does it work?” to “does it obey the developer’s intent?” For anyone building or deploying LLM‑powered coding assistants, the benchmark and the C2A Score provide a practical, industry‑ready way to assess and improve trustworthiness.

Authors

• Sravani Gunnu
• Shanmukha Guttula
• Hima Patel

Paper Information

• arXiv ID: 2512.17387v1
• Categories: cs.SE, cs.CL
• Published: December 19, 2025
• PDF: Download PDF