Kubernetes v1.35: Kubelet Configuration Drop-in Directory Graduates to GA

Source: Kubernetes Blog

What’s New?

• --config-dir flag is now fully supported.
• You can point the kubelet to a directory that contains multiple configuration files.
• All files in that directory are automatically merged with the main kubelet configuration.

Why Use a Drop‑In Directory?

• Centralised base configuration – keep a common set of settings for all kubelets.
• Targeted customisations – add node‑group‑specific tweaks without touching the base file.
• Simplified management – no need for complex tooling or manual edits per node.

How to Enable It

# Example kubelet systemd unit (or static pod) snippet
ExecStart=/usr/local/bin/kubelet \
  --config=/etc/kubernetes/kubelet-config.yaml \
  --config-dir=/etc/kubernetes/kubelet-config.d

• --config points to the primary configuration file (the base).
• --config-dir points to a directory (e.g., /etc/kubernetes/kubelet-config.d) containing additional YAML files.

Best Practices

Example Directory Layout

/etc/kubernetes/kubelet-config.d/
├── 01-base.yaml          # Common settings for all nodes
├── 10-gpu-nodes.yaml     # GPU‑specific flags (only applied to GPU nodes)
└── 20-custom-metrics.yaml# Extra metrics exporters

When the kubelet starts, it merges the files in lexical order (01‑base.yaml → 10‑gpu-nodes.yaml → 20‑custom-metrics.yaml) and applies the resulting configuration.

The Problem: Managing Kubelet Configuration at Scale

As Kubernetes clusters grow larger and more complex, they often include heterogeneous node pools with different hardware capabilities, workload requirements, and operational constraints. This diversity necessitates distinct kubelet configurations across node groups—yet managing these varied configurations at scale becomes increasingly challenging.

Key Pain Points

• Configuration drift – Slight differences between node configurations lead to inconsistent behavior.
• Node‑group customization – GPU nodes, edge nodes, and standard compute nodes each require their own kubelet settings.
• Operational overhead – Maintaining separate, complete configuration files for every node type is error‑prone and hard to audit.
• Change management – Rolling out configuration updates across heterogeneous pools demands careful coordination.

Prior Approaches

Before native support was added to Kubernetes, administrators had to choose one of the following, each with its own drawbacks:

1. Single monolithic configuration for all nodes – cannot accommodate node‑specific needs.
2. Manually maintained multiple full configs – high risk of drift and audit difficulty.
3. External tooling or scripts – adds complexity and often lacks integration with the control plane.

The New Solution

The graduation of kubelet configuration support to stable provides a fourth, fully supported way for cluster administrators to manage heterogeneous kubelet settings reliably and at scale.

Example Use Cases

Managing Heterogeneous Node Pools

Consider a cluster with multiple node types:

• Standard compute nodes
• High‑capacity nodes (e.g., GPUs or large memory)
• Edge nodes with specialized requirements

Base Configuration

File: 00-base.conf

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
  - "10.96.0.10"
clusterDomain: cluster.local

High‑Capacity Node Override

File: 50-high-capacity-nodes.conf

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 50
systemReserved:
  memory: "4Gi"
  cpu: "1000m"

Edge Node Override

File: 50-edge-nodes.conf – edge compute typically has lower capacity

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "5%"

Result:

• High‑capacity nodes load the base configuration plus the high‑capacity overrides.
• Edge nodes load the base configuration plus the edge‑specific overrides.

Gradual Configuration Rollouts

When rolling out configuration changes, follow this pattern:

1. Create a new drop‑in file with a high numeric prefix (e.g., 99-new-feature.conf).
2. Test the changes on a small subset of nodes.
3. Gradually expand the rollout to additional nodes.
4. Once the change is stable, merge the new settings into the base configuration.

Viewing the Merged Configuration

Since the configuration is now spread across multiple files, you can inspect the final merged configuration using the kubelet’s /configz endpoint:

# Start the kubectl proxy
kubectl proxy

# In another terminal, fetch the merged configuration
# Replace <node-name> with the name of your node
curl -X GET http://127.0.0.1:8001/api/v1/nodes/<node-name>/proxy/configz | jq .

The output shows the exact configuration the kubelet is using after all merging has been applied, including any settings supplied via kubelet command‑line arguments.

For detailed setup instructions, configuration examples, and merging behavior, see the official documentation:

• Set Kubelet Parameters via a Configuration File
• Kubelet Configuration Directory Merging

Good Practices

When using the kubelet configuration drop‑in directory:

• Test configurations incrementally – Always test new drop‑in configurations on a subset of nodes before rolling them out cluster‑wide to minimize risk.
• Version‑control your drop‑ins – Store your drop‑in configuration files (or the source that generates them) in version control alongside your infrastructure‑as‑code. This lets you track changes and roll back easily.
• Use numeric prefixes for predictable ordering – Name files with numeric prefixes (e.g., 00-, 50-, 90-) to explicitly control merge order and make the configuration layering obvious to other administrators.
• Be mindful of temporary files – Some text editors automatically create backup files (such as .bak, .swp, or files with a ~ suffix) in the same directory when editing. Ensure these temporary or backup files are not left in the configuration directory, as they may be processed by the kubelet.

Acknowledgments

This feature was developed through the collaborative efforts of SIG Node. Special thanks to all contributors who helped design, implement, test, and document the feature from its alpha release in v1.28, through beta in v1.30, to GA in v1.35.

Feedback & Discussion

• Join the Kubernetes Node Special Interest Group.
• Participate in the public Slack channel #sig-node.
• File an issue on GitHub.

Get Involved

If you have feedback or questions about kubelet configuration management, or want to share your experience using this feature, join the discussion:

• SIG Node community page – (link pending)
• Kubernetes Slack – join the #sig-node channel
• SIG Node mailing list – (link pending)

SIG Node would love to hear about your experiences using this feature in production!