How we solved cache invalidation in Kubernetes with a headless service

Source: Dev.to

🛑 The Problem: Fast Caching & Scalable Invalidation

We decided on in‑memory caching (using @nestjs/cache-manager) to slash DB load and boost response times. In a Kubernetes cluster, where the service runs across multiple, dynamic pods, this creates a major headache: cache inconsistency.

• If an admin updates a package price in MySQL, only the pod that handled the write sees the change immediately.
• All other pods continue serving stale data.

Additionally, pod IPs are temporary; they change during restarts, deployments, and scaling events. Hard‑coding IP lists is impossible, so we needed a reliable way to discover every running pod at the exact moment of a data update—a broadcasting mechanism.

🛠️ The Solution: The K8s Headless Service

Our strategy was a hybrid approach:

1. Local caching for performance.
2. Targeted internal HTTP broadcast for invalidation.

Step 1 – Headless Services for Pod Discovery

A standard Kubernetes Service acts as a load balancer with a single Virtual IP. A Headless Service (clusterIP: None) skips the load balancer and returns a list of individual DNS A records for every active pod’s IP address.

apiVersion: v1
kind: Service
metadata:
  name: booking-service-headless
spec:
  clusterIP: None               # THE TRICK: makes it headless
  selector:
    app: booking-service
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000

Now, resolving booking-service-headless.default.svc.cluster.local yields an array of all current, healthy pod IPs.

Visualization – Headless Services in action

Steps 2 & 3 – Broadcast Implementation

1. Add kubectl to the Docker image (or use the @kubernetes/client-node library) so the pod can query the Kubernetes API directly.
2. Apply the principle of least privilege: create a dedicated ServiceAccount and bind it to a Role that only allows list and get on the endpoints resource in its own namespace. This limits damage if a pod is compromised.
3. Expose an internal /invalidate-cache endpoint on the booking service. It receives a cache key and clears it using the in‑memory cache’s del() method.

When an admin update occurs, the pod that performed the write runs a script that:

• Retrieves the current pod IPs from the headless service endpoints.
• Sends an HTTP POST to each pod’s /invalidate-cache endpoint.

# Executed by the Node.js process on update
pod_ips=$(kubectl get endpoints booking-service-headless \
          -o jsonpath='{.subsets[0].addresses[*].ip}')

for ip in $pod_ips; do
  curl -X POST http://$ip:3000/v2/invalidate \
       -H "Content-Type: application/json" \
       -d '{"key":"package-123"}'   # Clears the specific key
done

Why this works

Alternative – If you prefer not to install kubectl/curl in the image, use the official @kubernetes/client-node library. It lets your Node.js code query the Kubernetes API directly, making the logic more testable, improving error handling, and removing the overhead of spawning shell processes.

📈 Production Results and Takeaways

After rollout:

• MySQL load stayed consistently low.
• Cache‑hit rate rose dramatically (≈ 95 % on package queries).
• Response latency dropped from ~200 ms to < 30 ms for cached requests.
• Operational overhead remained minimal—no new services, no extra cost, and the RBAC‑restricted ServiceAccount kept the security surface small.

Key lessons

1. Headless Services are a simple, built‑in discovery mechanism for pod‑level communication.
2. In‑memory caches + explicit invalidation give you the performance of a distributed cache without the operational complexity.
3. Least‑privilege ServiceAccounts protect your cluster even when you expose internal tooling inside containers.

If you’re dealing with mostly static data that still needs occasional updates, this pattern offers a low‑cost, Kubernetes‑native way to keep every pod’s cache in sync.

Result Summary

• Uptime: Remained above 95 %+.
• Incidents: Zero reported cases of users seeing stale package data.
• Headless Service: The trick proved robust, even during high‑traffic deployments.

Takeaway
This solution demonstrates that the most elegant and cost‑effective answer isn’t always an expensive managed service—it can be a creative application of the tools you already have. It’s about being smart with your infrastructure, not just throwing money at the problem.