Kubernetes Persistence Series Part 3: Controllers & Resilience — Why Kubernetes Self-Heals

Source: Dev.to

What You’ll Learn

• How application controllers (NGINX Ingress, cert‑manager) persist through evictions
• Why controllers are stateless and can restart anywhere
• The complete persistence chain from hardware to application
• What survives pod evictions vs. what doesn’t

---

Previously

• Part 1 – Debugged a missing ingress after GKE node upgrades.
• Part 2 – Explored how systemd supervises kubelet, and how kubelet bootstraps the control plane through static pods.

Now we reach the final layer: your application controllers—and the elegant insight that makes Kubernetes truly resilient.

Layer 4: Application Controllers

How Application Controllers Persist

Controllers such as NGINX Ingress, cert‑manager, and Prometheus Operator are typically deployed as Deployments or StatefulSets:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
  template:
    spec:
      containers:
        - name: controller
          image: registry.k8s.io/ingress-nginx/controller:v1.9.0

When this pod is evicted:

1. kubelet stops reporting the pod → the control plane marks it Terminated.
2. The ReplicaSet controller sees that currentReplicas (0) …
   Key point: The controller itself stores no state; it reads everything from the API server (backed by etcd).

Helm Release Persistence

Helm stores release information in Kubernetes Secrets:

kubectl get secret -n monitoring -l owner=helm -o yaml

apiVersion: v1
kind: Secret
metadata:
  name: sh.helm.release.v1.prometheus.v3
  labels:
    owner: helm
    name: prometheus
    version: "3"
type: helm.sh/release.v1
data:
  release: H4sIAAAAAAAAA...   # Base64‑encoded release manifest

The secret contains:

• The chart that was installed
• The values that were used
• The computed manifest of all resources

Because this secret is stored in etcd via the API server, Helm releases survive any pod eviction.

The Complete Persistence Chain

┌─────────────────────────────────────────────────────────────────────┐
│                     Linux Host (Physical/VM)                        │
├─────────────────────────────────────────────────────────────────────┤
│  systemd (PID 1)                                                    │
│  ├── Supervises all system services                                 │
│  ├── Restarts failed services automatically                         │
│  └── Config: /etc/systemd/system/                                   │
│      │                                                              │
│      └── kubelet.service                                            │
│          ├── Started and supervised by systemd                      │
│          ├── Watches /etc/kubernetes/manifests/ for static pods     │
│          ├── Watches API server for scheduled pods                  │
│          └── Ensures containers match pod specs                     │
│              │                                                      │
│              ├── Static Pods (/etc/kubernetes/manifests/)           │
│              │   ├── etcd ──────────────────┐                       │
│              │   ├── kube-apiserver ◄───────┤ Persistent            │
│              │   ├── kube-controller-manager│ State Store           │
│              │   └── kube-scheduler         │                       │
│              │                              │                       │
│              └── Regular Pods ◄─────────────┘                       │
│                  │                 (scheduled via API server)       │
│                  │                                                  │
│                  ├── kube-system namespace                          │
│                  │   ├── CoreDNS                                    │
│                  │   ├── kube-proxy                                 │
│                  │   └── CNI plugins                                │
│                  │                                                  │
│                  ├── ingress-nginx namespace                        │
│                  │   └── NGINX Ingress Controller                   │
│                  │       └── Watches Ingress resources              │
│                  │                                                  │
│                  └── Application namespaces                         │
│                      ├── cert-manager                               │
│                      ├── Prometheus Operator                        │
│                      └── Your applications                          │
└─────────────────────────────────────────────────────────────────────┘

The Critical Insight: Controllers Are Stateless

This is the elegant core of the design: controllers don’t store state.

The API server + etcd is the single source of truth, not the controllers themselves.

Supervision Hierarchy (Mermaid Diagram)

flowchart LR
    subgraph Source["Source of Truth"]
        etcd[(etcd)]
        api[API Server]
    end

    subgraph StatelessControllers["Stateless Controllers"]
        deploy[Deployment Controller]
        rs[ReplicaSet Controller]
        nginx[NGINX Ingress]
        cert[cert-manager]
    end

    etcd -->> api
    api -->> deploy
    api -->> rs
    api -->> nginx
    api -->> cert

    style etcd fill:#e67700,color:#fff
    style api fill:#e67700,color:#fff

(If the diagram does not render, view it directly on Mermaid Live Editor using the code above.)

Why This Matters

• Delete any controller pod → it restarts and catches up.
• Move controllers between nodes → they simply reconnect to the API server.
• Scale controllers to multiple replicas → they coordinate via the API server.
• Upgrade controllers → the new version reads the same state from etcd.

What Survives vs. What Doesn’t

Survives Any Pod Eviction

Doesn’t Survive Without Help

The Elegance of the Design

Kubernetes’ architecture follows several key design principles:

• Declarative over imperative – Describe the desired state, not the steps to get there.
• Reconciliation over transactions – Continuously converge toward the desired state.
• Stateless controllers – State lives in etcd, not in the controller processes.
• Hierarchical supervision – Each layer watches the layer above it.
• Failure is normal – Design for recovery, not for preventing failures.

Because of these principles, Kubernetes clusters can:

• Lose nodes unexpectedly.
• Have pods evicted due to resource pressure.
• Experience network partitions.
• Undergo rolling upgrades.

…and still maintain application availability.

Conclusion

The journey from debugging a missing Ingress to understanding the full supervision hierarchy reveals the sophisticated machinery that makes Kubernetes resilient.

systemd → kubelet → static pods → control plane → controllers → your apps

Each layer supervises the next, with etcd acting as the persistent memory that survives any component failure.

Key insight: Kubernetes doesn’t prevent failures—it recovers from them automatically through layered supervision, persistent state in etcd, and continuous reconciliation loops.

That is the true power of Kubernetes: not that things never fail, but that when they do, the system knows how to restore itself to the desired state.

Series Recap

• Part 1: When Our Ingress Vanished – The incident that started it all.
• Part 2: The Foundation – systemd → kubelet → control plane.
• Part 3: Controllers & Resilience – Why Kubernetes self‑heals.

Further Reading

• Kubernetes Components
• Static Pods
• Controller Manager
• systemd Service Files

Found this series useful? Follow for more Kubernetes internals content!