[Paper] Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing

Source: arXiv - 2602.10092v1

Overview

The paper “Quantum‑Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing” introduces a large‑scale benchmark that probes how well large language models (LLMs) actually understand quantum‑computing concepts—not just how well they can write code. By testing 26 state‑of‑the‑art models on 2,700 carefully crafted questions, the authors reveal surprising strengths and glaring blind spots that matter to developers, educators, and quantum‑tech startups alike.

Key Contributions

• A first‑of‑its‑kind benchmark (Quantum‑Audit) covering 2,700 questions on core quantum topics, including theory, algorithms, hardware, and security.
• Three question families:
  1. 1,000 expert‑written items (high‑quality, human‑curated).
  2. 1,000 LLM‑generated items extracted from recent research papers and validated by experts.
  3. 700 “challenge” items (350 open‑ended, 350 with deliberately false premises).
• Comprehensive evaluation of 26 leading LLMs, spanning open‑source and commercial offerings.
• Human baseline: 23 %–86 % accuracy across participants, with domain experts averaging 74 %.
• Key insight: Top commercial models (e.g., Claude Opus 4.5) can surpass expert averages on the overall benchmark but still stumble on expert‑written and security‑focused questions.
• Error‑propagation analysis: Models often accept false premises, achieving < 66 % accuracy on “detect‑the‑mistake” items.

Methodology

1. Question Design

• Expert‑written: Quantum researchers authored 1,000 multiple‑choice and short‑answer items covering fundamentals (qubits, superposition, measurement), algorithms (Grover, Shor), error correction, and emerging security concerns.
• LLM‑generated: A separate LLM scanned recent quantum‑computing papers, extracted statements, and turned them into questions. Human experts then vetted each item for correctness and relevance.
• Challenge set: Crafted to probe reasoning depth.
  • Open‑ended prompts ask the model to explain a concept or solve a problem without predefined options.
  • False‑premise items embed a subtle mistake (e.g., “If a qubit is measured in the X basis, its state collapses to |0⟩…”) and require the model to spot and correct it.

2. Model Evaluation

• Each model received the full 2,700‑question suite via a zero‑shot API call (no fine‑tuning).
• For multiple‑choice items, the model’s top‑ranked answer was compared to the ground truth.
• Open‑ended responses were graded by two independent quantum‑computing experts using a rubric that rewards correctness, completeness, and logical justification.

3. Human Baseline

• 30 participants (students, engineers, and quantum researchers) answered the same test under identical conditions. Their scores provide a reference for “reasonable” performance.

4. Metrics

• Primary metric: Accuracy (percentage of correctly answered items).
• Secondary analyses: performance split by question source (expert vs. LLM‑generated), difficulty tier (basic vs. advanced/security), and premise‑validation rate for false‑premise items.

Results & Findings

• Performance Gap: Even the top model loses ~12 points on expert‑written questions versus LLM‑generated ones, suggesting that curated, high‑quality prompts expose deeper reasoning gaps.
• Security Questions: Accuracy drops to the low‑70s, indicating that models are not yet reliable for nuanced topics like quantum cryptography or side‑channel attacks.
• Premise‑aware Reasoning: Models frequently accept incorrect assumptions, reinforcing them instead of flagging the error—a critical flaw for any advisory or tutoring system.

Practical Implications

• Educational Tools: LLMs can already serve as competent “first‑line” tutors for introductory quantum concepts, but developers should embed verification layers (e.g., cross‑checking with a knowledge base) before deploying them in formal curricula.
• Research Assistants: The high performance on LLM‑generated questions shows that models excel at summarizing and re‑phrasing recent literature, making them useful for rapid literature reviews—provided users remain vigilant about factual accuracy.
• Quantum‑Software Development: While code‑generation benchmarks remain strong, the reasoning deficits uncovered here warn against relying on LLMs for design reviews or security audits without human oversight.
• Product Roadmaps: Companies building quantum‑focused AI assistants should prioritize premise‑validation capabilities (e.g., built‑in logical consistency checks) to avoid the “hallucination‑reinforcement” problem highlighted by the false‑premise tests.
• Regulatory & Compliance: For sectors where quantum security is mission‑critical (finance, defense), the sub‑70 % accuracy on security questions suggests that current LLMs are not yet fit for autonomous decision‑making.

Limitations & Future Work

• Scope of Topics: The benchmark focuses on a curated set of core and emerging topics; ultra‑specialized areas (e.g., topological quantum error correction) remain untested.
• Zero‑Shot Setting: All models were evaluated without fine‑tuning; performance could improve with domain‑specific instruction tuning, which the authors plan to explore.
• Human Grading Subjectivity: Open‑ended answers were scored by experts, introducing potential bias; future releases will include a larger pool of annotators and inter‑rater reliability metrics.
• Dynamic Quantum Landscape: Quantum research evolves rapidly; maintaining the benchmark’s relevance will require periodic updates with new papers and emerging concepts.

Bottom line: Quantum‑Audit shines a light on where LLMs truly understand quantum computing—and where they merely sound plausible. For developers building the next generation of quantum‑aware AI tools, the findings are a call to combine the raw language prowess of LLMs with rigorous verification pipelines.

Authors

• Mohamed Afane
• Kayla Laufer
• Wenqi Wei
• Ying Mao
• Junaid Farooq
• Ying Wang
• Juntao Chen

Paper Information

• arXiv ID: 2602.10092v1
• Categories: cs.CL
• Published: February 10, 2026
• PDF: Download PDF