[Paper] Characterizing Bugs and Quality Attributes in Quantum Software: A Large-Scale Empirical Study

Source: arXiv - 2512.24656v1

Overview

The paper presents the first ecosystem‑wide, longitudinal study of bugs in quantum‑software projects. By mining 123 open‑source repositories spanning more than a decade, the authors reveal where defects arise, how they differ from classical bugs, and which quality attributes they threaten—offering a data‑driven roadmap for developers building reliable hybrid quantum‑classical systems.

Key Contributions

• Large‑scale dataset: 32,296 verified bug reports collected from 123 quantum‑software repositories (2012‑2024).
• Taxonomy of defects: A rule‑based classification that separates classical vs. quantum‑specific bugs and maps them to eight functional categories (full‑stack libraries, simulators, compilers, etc.).
• Defect density trends: Empirical evidence that defect density peaked between 2017‑2021 and has been declining as the ecosystem matures.
• Impact analysis: Quantifies how different bug types affect quality attributes such as performance, reliability, maintainability, and usability.
• Testing effectiveness: Shows that repositories with automated testing detect ~60 % fewer defects (negative‑binomial regression) and resolve issues faster.
• Actionable guidelines: Recommendations for testing, documentation, and maintenance practices tailored to quantum software.

Methodology

1. Repository selection: Identified 123 active open‑source quantum projects across eight functional domains (e.g., compilers, simulators, cryptography).
2. Data collection: Scraped issue trackers, commit histories, and static‑analysis reports; filtered to 32,296 verified bug reports (i.e., confirmed as defects).
3. Classification framework: Developed a validated rule‑based system that tags each bug as classical (e.g., UI, API misuse) or quantum‑specific (e.g., gate mis‑specification, noise‑model errors) and links it to a quality attribute.
4. Statistical analysis: Used descriptive statistics, longitudinal trend analysis, and a negative‑binomial regression to assess the relationship between automated testing and defect incidence.
5. Cross‑validation: Randomly sampled 10 % of the dataset for manual review to ensure classification accuracy (> 90 % agreement).

Results & Findings

• Most defect‑prone categories: Full‑stack libraries and compilers (≈ 38 % of bugs) – mainly due to circuit construction, gate mapping, and transpilation errors.
• Simulator bugs: Dominated by measurement handling and noise‑model inaccuracies, affecting simulation fidelity.
• Quality‑attribute impact:
  • Classical bugs → usability & interoperability issues.
  • Quantum‑specific bugs → severe degradation of performance, reliability, and maintainability.
• Severity hotspots: Cryptography, experimental computing, and compiler toolchains host the highest proportion of critical defects.
• Ecosystem maturation: Defect density rose sharply after 2015, peaked 2017‑2021, then fell ~ 22 % by 2024, indicating better tooling and developer experience.
• Testing payoff: Projects with CI‑driven automated tests reported 60 % fewer defects on average and closed issues 30 % faster than those without such pipelines.

Practical Implications

• Invest in CI/CD for quantum code: Automated test suites (unit, integration, and simulation‑based tests) are a proven lever to cut defect rates dramatically.
• Prioritize testing of compiler and library layers: Since these layers generate the bulk of bugs, adding regression tests for circuit generation, gate decomposition, and transpilation paths yields high ROI.
• Adopt quantum‑specific linting/static analysis: Tools that catch gate‑arity mismatches, invalid qubit indices, or improper noise‑model parameters can prevent the most damaging quantum bugs early.
• Documentation focus: Clear API contracts around quantum data structures (e.g., QuantumCircuit, QubitRegister) reduce classical usability bugs that often stem from ambiguous specifications.
• Performance‑aware debugging: Because quantum‑specific bugs disproportionately affect runtime and resource usage, integrating profiling (gate count, depth, error rates) into the test pipeline helps surface hidden performance regressions.
• Risk‑based triage: Teams working on cryptographic or experimental quantum algorithms should allocate extra QA resources, given the higher severity observed in those domains.

Limitations & Future Work

• Open‑source bias: The study only covers publicly available repositories; proprietary quantum stacks may exhibit different defect patterns.
• Classification granularity: While the rule‑based taxonomy achieved high agreement, some nuanced bugs (e.g., hybrid classical‑quantum race conditions) may be under‑represented.
• Tooling ecosystem evolution: Rapid changes in quantum SDKs (Qiskit, Cirq, Braket) could shift defect distributions; continuous monitoring is needed.
• Future directions: Extending the dataset to include private industry projects, refining automated classification with machine‑learning models, and evaluating the impact of emerging practices such as quantum‑aware fuzz testing and formal verification.

Authors

• Mir Mohammad Yousuf
• Shabir Ahmad Sofi

Paper Information

• arXiv ID: 2512.24656v1
• Categories: cs.SE
• Published: December 31, 2025
• PDF: Download PDF