AWS re:Invent 2025 - Building Zero-CVE Container Images at Scale: Patterns and Pitfalls (MAM215)
Source: Dev.to
Overview
AWS re:Invent 2025 – Building Zero‑CVE Container Images at Scale: Patterns and Pitfalls (MAM215)
In this session Dale Rodriguez (Chainguard) explains how Chainguard builds zero‑CVE container images at scale. The Chainguard factory rebuilds 1,818 images daily from source using tools such as Melange and APKO, processing with roughly 1,000 CPUs.
Key points
- Automated CVE monitoring with 7‑day SLA for critical issues and 14‑day SLA for high/medium/low severity.
- AI assists in diagnosing build errors, but all changes are verified by humans.
- Images undergo extensive testing on real Kubernetes clusters before release.
- Chainguard offers 54 free images and enterprise‑grade custom assembly options.
- The platform now also covers VMs and language libraries (Python, Java, JavaScript) with malware scanning and CVE back‑porting up to 3 years for packages.
This article is auto‑generated from the original presentation. Typos or minor inaccuracies may be present.
Inside the Chainguard Factory: Building Zero‑CVE Container Images Through Automation and Scale
Introduction
“My name is Dale Rodriguez, Sr. Happiness Engineer and Sr. Solutions Engineer at Chainguard.”
Chainguard’s mission is to be the secure source for open source—securing containers, libraries, and virtual machines so they can be run in production without compliance or security concerns.
- 1,818 images and 134,752 versions are maintained.
- The business model sells the images (and related services), not the factory itself.
Toolchain
| Tool | Purpose |
|---|---|
| Melange | Package manager that builds APKO packages |
| APKO | Creates OCI images from packages |
| Cosign | GitHub security tool for signing images |
| Sigstore | Provides signing infrastructure |
| Terraform providers | Infrastructure as code for the factory |
| Malcontent | Scans container images for malware |
Zero‑CVE results from heavy automation, extensive debugging, large compute resources, thorough testing, and continuous validation.
Philosophy of Building from Source
- Trust the source – Building from source gives visibility into every component, reducing blind spots.
- Depth of defense – Expanding visibility down to the file‑system level mitigates uncertainty and improves trust.
- Safer compiler options – Re‑compiling upstream packages with hardened compiler flags reduces vulnerabilities.
The Chainguard Factory Workflow
- Download OSS project – source code is fetched.
- Automated package build – Melange creates packages; APKO assembles images.
- CVE remediation – If a vulnerability is detected, an automated patch is applied.
- Package repository – Built packages are stored.
- Image build & testing – Images are built, then tested on real Kubernetes clusters.
- Push to image repository – Verified images are published for consumption.
All 1,818 projects are rebuilt daily to address zero‑day CVEs and keep dependencies up‑to‑date.
Assembly Line
The factory’s assembly line is the central pipeline; every component passes through it. Continuous daily rebuilds ensure:
- Immediate response to newly disclosed CVEs.
- Consistent updates aligned with upstream project releases.
Media
