[Paper] A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM
Published: (January 9, 2026 at 03:26 AM EST)
4 min read
Source: arXiv
Source: arXiv - 2601.05622v1
Overview
The paper presents the first large‑scale, two‑stage empirical study of how well today’s Software Bill‑of‑Materials (SBOM) tools actually follow the official SBOM standards (e.g., SPDX, CycloneDX). By automatically generating and analysing more than 55 k SBOMs from real‑world open‑source projects, the authors expose a substantial “adherence gap” that can jeopardise supply‑chain security and compliance workflows.
Key Contributions
- SAP Framework – an open‑source, automated evaluation platform that parses SBOMs, checks them against standard policy rules, and computes cross‑tool consistency metrics.
- Comprehensive Dataset – 55 444 SBOMs produced by six popular SBOM generators across 3 287 distinct repositories, collected at two points in time (baseline and one‑year later).
- Empirical Findings – quantitative evidence of (1) weak policy compliance, (2) severe inter‑tool and intra‑tool consistency problems, and (3) low accuracy for critical fields such as package licenses.
- Root‑Cause Analysis – identification of missing standard features, ambiguous spec wording, and implementation shortcuts that drive the observed gaps.
- Practical Recommendations – concrete fixes for tool developers (e.g., stricter schema validation, deterministic component ordering) and guidance for organizations adopting SBOMs.
- Open‑Source Release – all code, Docker images, and raw results are publicly available for replication and further research.
Methodology
- Tool Selection – Six widely‑used SBOM generators (covering SPDX and CycloneDX output) were chosen based on GitHub stars, download counts, and language coverage.
- Repository Corpus – 3 287 open‑source projects spanning Java, JavaScript, Python, Go, Rust, and C/C++ were cloned from GitHub, ensuring a realistic mix of build systems and dependency managers.
- Two‑Stage Evaluation
- Baseline – each tool generated an SBOM for every repository; the SAP framework validated the SBOMs against the corresponding standard’s policy rules (required fields, data types, SPDX license identifiers, etc.).
- Longitudinal Follow‑up – after 12 months, the same tools were run on the same repository set (using the latest released versions) to assess stability over time.
- Metrics – compliance rate (percentage of SBOMs satisfying all mandatory rules), inter‑tool consistency (Jaccard similarity of detected packages across tools), intra‑tool consistency (change in output between versions), and field‑level accuracy (e.g., license detection compared to a manually curated ground truth).
- Root‑Cause Investigation – for each major deficiency, the authors inspected tool source code, issue trackers, and the standards themselves to pinpoint why the gap existed.
Results & Findings
| Aspect | Key Result | Interpretation |
|---|---|---|
| Policy Compliance | Only 31 % of SBOMs met all mandatory standard fields; many omitted required SPDX license identifiers or component hashes. | Tools often generate “minimal” SBOMs that are technically parsable but insufficient for downstream compliance checks. |
| Inter‑Tool Consistency | Package detection overlap ranged from 7.84 % (Go) to 12.77 % (JavaScript) across tools. | Different parsers and dependency‑graph algorithms lead to wildly divergent component lists, making cross‑tool validation unreliable. |
| Intra‑Tool Consistency | Same tool, new version vs. old version showed ≤ 15 % overlap for many repositories. | Updates introduce regressions or change heuristics, breaking reproducibility of SBOMs over time. |
| License Accuracy | Correct license attribution was below 20 % for most tools; many reported “UNKNOWN” or mis‑identified licenses. | License detection logic is under‑engineered, which undermines one of the primary security use‑cases of SBOMs. |
| Root Causes | – Ambiguous spec sections (e.g., optional vs. required fields). – Lack of deterministic ordering in component lists. – Dependency‑graph extraction tools (e.g., npm ls, go mod graph) produce inconsistent outputs. | These systemic issues can be addressed by tighter spec‑tool alignment and more robust parsing pipelines. |
Practical Implications
- For DevOps Teams – Relying on a single SBOM generator may give a false sense of security; cross‑checking with at least one alternative tool is advisable, especially before compliance audits.
- For Tool Vendors – Implement full schema validation, deterministic component ordering, and expose version‑stable APIs to avoid breaking downstream pipelines after upgrades.
- For Security Platforms – Integrate SBOM validation as a first‑class step (e.g., CI/CD gate) to catch missing fields or malformed licenses before they propagate to vulnerability scanners.
- For Standards Bodies – The findings highlight ambiguous language in SPDX/CycloneDX that needs clarification; tighter conformance test suites could improve ecosystem consistency.
- For Supply‑Chain Automation – Accurate license data and stable component lists are prerequisites for automated policy enforcement (e.g., “no GPL‑licensed dependencies”). The current low accuracy suggests that many automated policy tools are operating on noisy inputs, leading to false positives/negatives.
Limitations & Future Work
- Scope of Languages – The study covered six popular languages; niche ecosystems (e.g., .NET, Swift) were not evaluated and may exhibit different adherence patterns.
- Ground‑Truth Construction – License labels were manually curated for a subset of packages; scaling this to the entire corpus could reveal additional nuances.
- Tool Set – Only six generators were examined; newer or commercial SBOM solutions might behave differently.
- Dynamic Dependencies – The analysis focused on static dependency graphs; runtime‑generated components (e.g., plugins loaded at execution) remain unaddressed.
Future research directions include expanding the language coverage, building a community‑driven conformance test suite for SBOM standards, and investigating how SBOM inaccuracies affect downstream vulnerability‑management tools in real production pipelines.
Authors
- Chengjie Wang
- Jingzheng Wu
- Hao Lyu
- Xiang Ling
- Tianyue Luo
- Yanjun Wu
- Chen Zhao
Paper Information
- arXiv ID: 2601.05622v1
- Categories: cs.SE
- Published: January 9, 2026
- PDF: Download PDF