[Paper] TaskEval: Synthesised Evaluation for Foundation-Model Tasks

Source: arXiv - 2512.04442v1

Overview

The paper TaskEval: Synthesised Evaluation for Foundation‑Model Tasks tackles a pain point that many dev teams hit when building applications on top of large foundation models (LLMs, multimodal models, etc.): how do you reliably test whether the model is doing the right thing for your very specific use case? The authors present a system that automatically generates a custom evaluator—complete with a lightweight UI for human feedback—so you can assess model outputs even when no off‑the‑shelf benchmark or metric exists.

Key Contributions

• Task‑agnostic meta‑model that encodes the essential properties of any FM‑driven task (inputs, expected outputs, constraints).
• Interaction protocol that blends automated checks with targeted human feedback, minimizing the amount of manual review needed.
• Eval synthesiser that either selects from an existing library of evaluation primitives or generates new ones on‑the‑fly, tailoring the evaluation suite to the task at hand.
• Tool implementation (TaskEval) demonstrated on two real‑world scenarios: extracting data from charts and answering questions over documents.
• Empirical validation showing that the synthesized evaluators achieve 93 % and 90 % accuracy on the two case studies, respectively.

Methodology

1. Meta‑model construction – The authors first define a generic schema that captures what a task looks like (e.g., input type, output type, correctness criteria). This schema is deliberately lightweight so it can be populated by a developer in a few minutes.
2. Human‑in‑the‑loop protocol – Instead of asking engineers to label thousands of examples, TaskEval asks for strategic feedback. The system proposes a small set of representative inputs, the developer judges the model’s responses, and the feedback is used to refine the evaluator.
3. Eval synthesis – With the meta‑model and feedback in hand, an internal “synthesiser” either (a) pulls a matching evaluator from a curated library (e.g., BLEU for translation‑style outputs) or (b) composes a new evaluator by stitching together primitive checks (format validation, numeric tolerance, logical consistency).
4. Deployment – The generated evaluator runs automatically as part of the CI/CD pipeline, while the UI surface lets developers inspect failures and provide additional hints when needed.

Results & Findings

• Chart data extraction – TaskEval produced a custom evaluator that checks column headings, numeric ranges, and visual‑to‑text alignment. In a held‑out test set, the evaluator correctly flagged 93 % of hallucinated or mis‑extracted entries.
• Document question answering – For a QA system over PDFs, the synthesized evaluator combined answer span extraction with citation verification. It achieved 90 % accuracy in spotting incorrect answers.
• Human effort reduction – The interaction protocol required roughly 5–10 minutes of developer feedback per task, a drastic cut from the hours typically spent curating a benchmark dataset.

Practical Implications

• Plug‑and‑play evaluation – Teams can spin up a task‑specific test suite without hunting for a public benchmark that matches their niche use case.
• CI/CD safety net – The generated evaluators can be integrated into automated testing pipelines, catching hallucinations before they reach production.
• Rapid prototyping – When experimenting with new prompts or model variants, developers get immediate, quantitative feedback on whether the change actually improves task performance.
• Cost savings – By limiting the need for large labeled test sets, companies can allocate budget to model fine‑tuning or data collection where it matters most.

Limitations & Future Work

• Scope of meta‑model – While designed to be task‑agnostic, the current schema may still struggle with highly interactive or multi‑turn tasks (e.g., code generation with iterative debugging).
• Evaluation granularity – The synthesized evaluators focus on binary correctness; richer metrics (e.g., partial credit, confidence calibration) are not yet supported.
• User study size – The paper reports preliminary results on two tasks; broader validation across more domains (e.g., code synthesis, multimodal reasoning) is needed to confirm generality.
• Future directions include expanding the primitive evaluator library, automating meta‑model extraction from API specifications, and exploring active‑learning loops that continuously improve the evaluator as the underlying FM evolves.

Authors

• Dilani Widanapathiranage
• Scott Barnett
• Stefanus Kurniawan
• Wannita Takerngsaksiri

Paper Information

• arXiv ID: 2512.04442v1
• Categories: cs.AI, cs.SE
• Published: December 4, 2025
• PDF: Download PDF