Kubernetes Deep Dive: Kubelet

Source: Dev.to

Introduction

In a Kubernetes cluster, the Kubelet is responsible for starting and stopping containers and managing the state of the node. It is a standalone binary that runs directly on the node’s operating system, typically started as a systemd service. In development environments such as kind, the node itself runs inside a Docker container, and kubelet is launched as a child process of the container’s init system.

When the kubelet stops, the control plane marks the node as NotReady, preventing new workloads from being scheduled there.

Kubelet Responsibilities

Kubelet continuously reconciles the actual containers on the node with the desired state defined in the PodSpec obtained from the API server. It does this through several specialized managers that operate autonomously:

• Pod Lifecycle Management – Handles pod creation, updates, and deletion via the SyncPod logic.
• Storage Bridge – The VolumeManager mounts external storage on the host and makes it available to containers.
• Secret Delivery – TokenManager and SecretManager fetch ServiceAccount tokens and secrets, injecting them into the appropriate locations.
• Status Reporting – Periodically pushes node resource usage and pod health status to the API server.
• Health Checks – Executes liveness and readiness probes, restarting containers when necessary.
• Node Self‑Defense – Performs pod eviction when resource limits are approached, protecting the node from a total crash.

All of these components coordinate around a central event loop known as the SyncLoop.

Runtime Stack: CRI → containerd → runc

When kubelet issues a “create container” command, the request flows through a layered stack:

1. CRI (Container Runtime Interface) – The protocol kubelet uses to communicate with the container runtime.
2. containerd (CRI Runtime) – Receives CRI commands, handles image pulls, and manages higher‑level container lifecycle operations.
3. runc (OCI Runtime) – Interacts directly with the kernel to create and run the container process.

Kubelet instructs containerd, which in turn invokes runc to perform the low‑level actions.

Eviction vs. OOM Killer

Pods can be terminated for memory shortage due to either:

• Violation of their own limits – the pod exceeds the resources it requested.
• Node‑wide crisis – the entire node is under pressure.

Kubelet aims to perform organized eviction before the kernel’s OOM Killer intervenes.

SyncPod Flow (Pod Startup)

1. Context Preparation – Retrieve the latest pod definition from the API server.
2. Volume Mount – Instruct VolumeManager to mount required storage on the host.
3. Secret/ConfigMap Preparation – TokenManager fetches service account tokens and places them as files on the host.
4. Sandbox Creation – Create the “pause” container (RunPodSandbox) that provides the pod’s network and namespace frame.
5. Container Startup –
   • Bind‑mount the prepared token or volume into the container’s filesystem (e.g., /var/run/secrets/...).
   • Launch the application container.

Summary

Kubelet is more than a simple agent; it is an autonomous control system that:

• Runs as a process on the host (or inside a kind node container).
• Translates high‑level pod specifications into concrete containers using the CRI → containerd → runc stack.
• Defends node health by evicting pods before the kernel’s OOM Killer can cause abrupt failures.

Understanding this “brain” of the node is essential when troubleshooting pod deaths (OOMKilled vs. Evicted) or volume‑mount issues.