AWS re:Invent 2025 - Secure Amazon ECS observability with CDK and Grafana (DEV338)
Source: Dev.to
Overview
AWS re:Invent 2025 – Secure Amazon ECS observability with CDK and Grafana (DEV338)
In this session Chibuike Nwachukwu presents a secure observability solution for Amazon ECS built with the AWS Cloud Development Kit (CDK) and Grafana. The architecture addresses the challenge of enabling non‑technical healthcare staff to monitor application logs without relying on CloudWatch. Key components include:
- AWS Client VPN for secure remote access
- ADOT sidecars to collect metrics and traces, sending them to AWS X‑Ray and Prometheus
- FireLens to forward logs to Loki
- Grafana dashboards for visualizing logs, metrics, and traces
The solution follows a security‑first design using private subnets, leverages OpenTelemetry for vendor‑neutral instrumentation, and automates deployment with CDK and GitHub Actions. The demo showcases auto‑instrumentation, custom metric collection, and error tracing through Grafana, illustrating how a simulated 404 error in DynamoDB operations is identified and diagnosed.
The Challenge: Observability for Non‑Technical Users
The healthcare startup’s non‑technical staff needed a way to view live application activity—sign‑ups, doctor calls, and error events—without navigating CloudWatch. They required an intuitive interface that presented logs, metrics, and traces in a clear, self‑service manner.
Architecture Overview
- Amazon ECS runs the application containers.
- ADOT (AWS Distro for OpenTelemetry) sidecars collect metrics and traces, exporting to Prometheus (for metrics) and AWS X‑Ray (for traces).
- FireLens sidecar forwards container logs to Loki.
- Grafana visualizes data from Loki, Prometheus, and X‑Ray, providing dashboards accessible via the VPN.
- AWS Client VPN secures remote access, keeping all traffic within private subnets.
Observability Foundations
Observability is the ability to infer the internal state of a system from its external outputs (logs, metrics, traces). By aggregating these signals in a unified platform, teams can:
- Detect anomalies quickly.
- Correlate events across services.
- Diagnose issues without deep technical knowledge.
Grafana’s interactive dashboards make these signals approachable for non‑technical stakeholders.
Core Pillars of the Solution
Security
- Private subnets isolate ECS tasks.
- AWS Client VPN provides encrypted access.
- IAM roles and security groups enforce least‑privilege access to observability resources.
Infrastructure as Code (CDK)
- All resources (VPC, ECS cluster, ADOT sidecars, FireLens, Loki, Grafana) are defined in CDK stacks.
- GitHub Actions automate CI/CD, ensuring reproducible deployments.
Grafana & Loki
- Grafana dashboards present logs (Loki), metrics (Prometheus), and traces (X‑Ray) in a single view.
- Loki stores logs cost‑effectively, while Grafana’s query editor lets users explore data without writing code.
Demo Highlights
- Auto‑instrumentation of ECS services using OpenTelemetry.
- Custom metrics (e.g., request latency) collected via ADOT sidecars.
- Error tracing: a simulated DynamoDB 404 error appears in Grafana, linking the log entry, metric spike, and trace span.
Further Resources
- AWS CDK Documentation – https://docs.aws.amazon.com/cdk/latest/guide/home.html
- Grafana Loki – https://grafana.com/oss/loki/
- OpenTelemetry – https://opentelemetry.io/
- AWS Distro for OpenTelemetry (ADOT) – https://aws.amazon.com/otel/
Video
Video source: AWS re:Invent 2025 – Secure Amazon ECS observability with CDK and Grafana (DEV338) (thumbnails linked throughout the article).