Project: One App — Three Probes — Real Failures

Source: Dev.to

Visual mental model (keep this in mind first)

Traffic rule (critical)

• ❌ Startup probe FAIL → container is NOT checked by liveness/readiness probes.
• ❌ Readiness probe FAIL → pod gets NO traffic.
• ❌ Liveness probe FAIL → pod is RESTARTED.

What you will demonstrate

You will run one app that:

• Starts slowly (startup probe needed)
• Can become temporarily unavailable (readiness probe needed)
• Can become stuck (liveness probe needed)

This mirrors real production behavior.

Step 0 — Prerequisites

minikube start
kubectl get nodes

Step 1 — Demo application (intentionally imperfect)

We’ll use a small HTTP app that:

• Takes 20 seconds to start
• Exposes /ready and /health endpoints

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: probe-demo
  template:
    metadata:
      labels:
        app: probe-demo
    spec:
      containers:
      - name: app
        image: ghcr.io/stefanprodan/podinfo:6.5.3
        ports:
        - containerPort: 9898

Apply the manifest:

kubectl apply -f deployment.yaml
kubectl get pods

At this point:

• The app starts slowly.
• Kubernetes does not know that traffic could hit it too early.

Step 2 — Expose the app

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: probe-demo-svc
spec:
  selector:
    app: probe-demo
  ports:
  - port: 80
    targetPort: 9898
  type: NodePort

Create the service and open it in the browser:

kubectl apply -f service.yaml
minikube service probe-demo-svc

Now the real learning starts.

Startup Probe

What it is

“Don’t judge this container until it has fully started.”

Why DevOps needs it

Without a startup probe, the liveness probe may restart slow apps endlessly (e.g., JVM, Python, DB‑connected apps).

Add a startup probe

startupProbe:
  httpGet:
    path: /health
    port: 9898
  failureThreshold: 30   # try for 30 seconds
  periodSeconds: 1

Meaning:
Kubernetes waits up to 30 seconds before marking the container as started. During this period, liveness and readiness probes are disabled.

Demonstrate failure (remove the probe)

kubectl delete pod -l app=probe-demo
kubectl describe pod 

What you’ll see:

• The pod restarts before finishing startup → classic CrashLoopBackOff for slow apps.

Readiness Probe

What it is

“Can this pod receive traffic right now?”

Why DevOps cares

• Prevents traffic during deploys.
• Stops broken pods from serving users.
• Required for zero‑downtime rolling updates.

Add a readiness probe

readinessProbe:
  httpGet:
    path: /ready
    port: 9898
  initialDelaySeconds: 5
  periodSeconds: 3

Observe

kubectl get pods
kubectl describe pod 
kubectl get endpoints probe-demo-svc

When readiness fails:

• The pod stays Running.
• It is removed from Service endpoints → no traffic is sent.

Demonstrate the issue if missing

Remove the readiness probe and deploy a new version → traffic may hit half‑started pods, resulting in 502 / empty responses for users.

Liveness Probe

What it is

“Is this container still healthy or stuck?”

Why DevOps needs it

• Apps can hang forever (deadlocks, memory leaks).
• Kubernetes must restart them automatically.

Add a liveness probe

livenessProbe:
  httpGet:
    path: /health
    port: 9898
  initialDelaySeconds: 20
  periodSeconds: 5

Demonstrate automatic restart

Simulate a failure:

kubectl exec -it <pod-name> -- kill 1

Observe the result:

kubectl get pods

You will see that the pod restarts automatically—no manual intervention required.

Final combined (production‑correct)

startupProbe:
  httpGet:
    path: /health
    port: 9898
  failureThreshold: 30
  periodSeconds: 1

readinessProbe:
  httpGet:
    path: /ready
    port: 9898
  periodSeconds: 3

livenessProbe:
  httpGet:
    path: /health
    port: 9898
  initialDelaySeconds: 20
  periodSeconds: 5

DevOps Engineer Checklist (Interview‑Level)

You must know:

• Startup → protects slow‑boot apps
• Readiness → controls traffic
• Liveness → auto‑recovery

Common pitfalls:

• Readiness ≠ Liveness (most common mistake)
• Missing readiness → downtime
• Missing liveness → silent failure
• Wrong startup configuration → endless restarts

How to explain this in production terms

“Startup protects initialization, readiness protects users, liveness protects the platform.”