[Paper] Many Tools, Few Exploitable Vulnerabilities: A Survey of 246 Static Code Analyzers for Security

Source: arXiv - 2602.18270v1

Overview

Static code analysis tools promise to catch security bugs before they ship, but how well do they actually cover the threat landscape?
Hermann, Peldszus, and Berger surveyed 246 publicly available static security analyzers, mapping what weaknesses they target, the domains they serve, and how they’re evaluated. Their findings reveal a surprising mismatch: most tools focus on a narrow set of vulnerabilities, and the flaws they flag are rarely exploitable in practice.

Key Contributions

• Comprehensive catalog of 246 static security analyzers, the largest systematic review of its kind.
• Taxonomy of targeted vulnerability classes, application domains, and underlying analysis techniques (e.g., data‑flow, symbolic execution, pattern matching).
• Critical assessment of evaluation practices, exposing a reliance on tiny, custom benchmarks that hinder reproducibility.
• Empirical insight that the majority of reported findings correspond to low‑severity or non‑exploitable issues.
• Guidelines for researchers and tool vendors on improving coverage, evaluation rigor, and real‑world relevance.

Methodology

1. Literature & Tool Collection – Systematic searches across academic databases, conference proceedings, and open‑source repositories, applying inclusion criteria (publicly available, security‑focused static analysis).
2. Classification Framework – Each tool was annotated for:
   • Targeted weaknesses (e.g., buffer overflows, injection, cryptographic misuse).
   • Supported languages & platforms.
   • Analysis technique (static taint analysis, abstract interpretation, model checking, etc.).
   • Evaluation method (benchmark suites, case studies, manual inspection).
3. Data Synthesis – Aggregated the annotations to produce quantitative distributions (e.g., % of tools covering OWASP Top‑10 categories) and qualitative observations about evaluation quality.
4. Validity Checks – Cross‑checked a random subset of entries with tool documentation and, where possible, ran the tools on a small sample codebase to verify reported capabilities.

Results & Findings

What this means: The static analysis market is fragmented, with many tools offering overlapping, shallow coverage. Developers may be lulled into a false sense of security, especially when tools flag many non‑exploitable issues that drown out the truly critical bugs.

Practical Implications

• Tool Selection: Prioritize analyzers that employ deeper semantic techniques (e.g., taint tracking, symbolic execution) and that have demonstrated coverage beyond the “low‑hanging fruit.”
• Integration Strategy: Combine multiple complementary tools (e.g., a lightweight rule‑based scanner for quick CI feedback plus a heavyweight analyzer for nightly deep scans) to broaden vulnerability detection without overwhelming developers with noise.
• Benchmarking & CI: Adopt standardized, community‑maintained benchmark suites (e.g., Juliet, SARD, or the newer CodeQL dataset) to evaluate tool efficacy before committing to a purchase or open‑source adoption.
• Risk Prioritization: Use the paper’s insight that most findings are non‑exploitable to calibrate alert thresholds, focusing triage effort on high‑severity, CVE‑linked warnings.
• Vendor Feedback Loop: Encourage vendors to publish transparent evaluation data and to update rule sets regularly, especially for newer languages and modern libraries.

Limitations & Future Work

• Scope of Tools: The survey only includes tools with publicly available documentation; proprietary or internal enterprise scanners may exhibit different characteristics.
• Depth of Evaluation: The authors relied on reported benchmark results rather than re‑executing every tool on a common dataset, which could mask hidden performance or precision issues.
• Evolving Landscape: Static analysis techniques evolve rapidly; a follow‑up study will be needed to track adoption of AI‑augmented analysis (e.g., large‑language‑model‑based scanners) and coverage of newer language ecosystems.

Bottom line: While static code analyzers remain a cornerstone of secure development pipelines, this extensive survey warns that quantity does not equal quality. Developers and security teams should be discerning about which tools they trust, how they evaluate them, and how they integrate findings into a broader, risk‑aware security strategy.

Authors

• Kevin Hermann
• Sven Peldszus
• Thorsten Berger

Paper Information

• arXiv ID: 2602.18270v1
• Categories: cs.CR, cs.SE
• Published: February 20, 2026
• PDF: Download PDF