MINI PROJECT: Headless Service with StatefulSet (MySQL-style behavior)

Source: Dev.to

🎯 Goal

By the end you will clearly see:

• Why ClusterIP hides pod identity
• Why Headless Service exposes pod identity
• How StatefulSet + Headless Service gives stable DNS per pod (the magic needed for databases)

🧠 Mental model (keep this in mind)

🧩 Project structure

headless-demo/
├── mysql-headless.yaml
├── mysql-statefulset.yaml
└── dns-test.yaml

1️⃣ Headless Service (NO ClusterIP)

File: mysql-headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None            # 👈 THIS MAKES IT HEADLESS
  selector:
    app: mysql
  ports:
    - port: 3306

Important:

• No virtual IP is created.
• DNS will return the pod IPs.

2️⃣ StatefulSet (stable pod identity)

File: mysql-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql-headless   # 👈 REQUIRED
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: root
          ports:
            - containerPort: 3306

What StatefulSet guarantees

mysql-0
mysql-1
mysql-2

• Names never change
• Identity is stable

3️⃣ DNS test pod (to observe behavior)

File: dns-test.yaml

apiVersion: v1
kind: Pod
metadata:
  name: dns-test
spec:
  containers:
    - name: dns
      image: busybox:1.28
      command: ["sleep", "3600"]

4️⃣ Apply everything (order matters)

kubectl apply -f mysql-headless.yaml
kubectl apply -f mysql-statefulset.yaml
kubectl apply -f dns-test.yaml

Wait for the pods to be ready:

kubectl get pods

You should see:

mysql-0   Running
mysql-1   Running
mysql-2   Running
dns-test  Running

5️⃣ 🔥 THE MOST IMPORTANT PART — DNS BEHAVIOR

Enter the test pod

kubectl exec -it dns-test -- sh

DNS lookup of the headless service

nslookup mysql-headless

Result – multiple IPs (one per pod)

Address: 10.244.0.12
Address: 10.244.0.13
Address: 10.244.0.14

This is DNS round‑robin.

DNS lookup of individual pods (the key)

nslookup mysql-0.mysql-headless
nslookup mysql-1.mysql-headless
nslookup mysql-2.mysql-headless

Each command resolves to a specific pod IP – something a plain ClusterIP service can never do.

6️⃣ Why databases need this

• Writes → mysql-0.mysql-headless (stable primary)
• Reads → replicas
• Replication → stable target

Impossible with a Deployment + ClusterIP because the service hides pod identities.

7️⃣ Visual intuition (what’s happening)

8️⃣ One‑line interview answer (remember this)

“A headless service removes the virtual IP and exposes pod identities via DNS. Combined with StatefulSets, it provides stable per‑pod DNS names, which is required for databases and leader‑follower architectures.”

🔥 SAME APP, TWO SERVICES

🟦 CASE 1 — ClusterIP Service (default behavior)

YAML (normal service)

apiVersion: v1
kind: Service
metadata:
  name: mysql-clusterip
spec:
  selector:
    app: mysql
  ports:
    - port: 3306

What Kubernetes does

• Creates ONE virtual IP
• Hides all pod IPs
• kube-proxy load‑balances traffic

DNS behavior inside the cluster

nslookup mysql-clusterip

Name:    mysql-clusterip
Address: 10.96.120.15    mysql-0
   +--> mysql-1
   +--> mysql-2

Kubernetes decides which pod receives each request.

❌ Why this breaks databases

If a write lands on any replica (e.g., mysql-2), the primary may not see it, breaking consistency and replication. A headless service with stable pod DNS avoids this problem.

❌ Problem: Login Fails

- Login request → mysql-0 (no data)
- ❌ login fails

Why?
Write and read traffic hit different pods. Pod identity is hidden, so you can’t force traffic to always go to mysql-0.

ClusterIP is stateless‑friendly but stateful‑hostile.

🟨 CASE 2 — Headless Service (NO CLUSTER IP)

We change only one line in the Service definition.

YAML (headless service)

apiVersion: v1
kind: Service
metadata:
  name: mysql-headless
spec:
  clusterIP: None   # 👈 THIS IS EVERYTHING
  selector:
    app: mysql
  ports:
    - port: 3306

🔍 DNS behavior (key difference)

Inside a pod:

nslookup mysql-headless

Result

Address: 10.244.0.10   (mysql-0)
Address: 10.244.0.11   (mysql-1)
Address: 10.244.0.12   (mysql-2)

• ⚠️ No virtual IP.
• ⚠️ DNS returns pod IPs directly – a DNS round‑robin, not kube‑proxy load balancing.

🧠 Still not enough alone (important)

If you stop here and use just mysql-headless, your app may still hit different pods because DNS answers rotate.
Headless alone is not the full solution.

🟩 ENTER STATEFULSET (THE MISSING PIECE)

A StatefulSet guarantees stable pod names:

mysql-0
mysql-1
mysql-2

Kubernetes automatically creates DNS records for each pod:

mysql-0.mysql-headless
mysql-1.mysql-headless
mysql-2.mysql-headless

🔥 The moment it clicks

Write traffic

mysql-0.mysql-headless

Read traffic

mysql-1.mysql-headless
mysql-2.mysql-headless

No guessing. No randomness.

📊 SIDE‑BY‑SIDE SUMMARY (MEMORIZE THIS)

🧪 Why we used a dns-test pod

You cannot “see DNS” from your laptop.
Create a pod just to run:

nslookup 

That’s how SREs debug real production issues.

🎯 ONE‑LINE INTERVIEW ANSWER (IMPORTANT)

“ClusterIP services hide pod identity and load‑balance traffic, which is unsuitable for databases.
Headless services expose pod IPs via DNS, and when combined with StatefulSets, provide stable per‑pod DNS required for stateful workloads.”

Images