Kubernetes 1.35: Enhanced Debugging with Versioned z-pages APIs

Source: Kubernetes Blog

What are z‑pages?

z‑pages are special debugging endpoints exposed by Kubernetes control‑plane components. Introduced as an alpha feature in Kubernetes 1.32, these endpoints provide runtime diagnostics for components such as:

• kube-apiserver
• kube-controller-manager
• kube-scheduler
• kubelet
• kube-proxy

The name “z‑pages” comes from the convention of using /*z paths for debugging endpoints.

Current z‑page endpoints

These endpoints are valuable for human operators who need to quickly inspect component state, but until now they only returned plain‑text output that was difficult to parse programmatically.

What’s new in Kubernetes 1.35?

Kubernetes 1.35 introduces structured, versioned responses for both /statusz and /flagz. The enhancement:

• Maintains backward compatibility – plain‑text is still the default.
• Adds optional JSON output – machine‑readable and versioned.

Backward‑compatible design

The new structured responses are opt‑in. Without an Accept header the endpoints continue to return the familiar plain‑text format:

$ curl --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
       --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
       --cacert /etc/kubernetes/pki/ca.crt \
       https://localhost:6443/statusz
kube-apiserver statusz
Warning: This endpoint is not meant to be machine parseable, has no formatting compatibility guarantees and is for debugging purposes only.
Started: Wed Oct 16 21:03:43 UTC 2024
Up: 0 hr 00 min 16 sec
Go version: go1.23.2
Binary version: 1.35.0-alpha.0.1595
Emulation version: 1.35
Paths: /healthz /livez /metrics /readyz /statusz /version

Structured JSON responses

To receive a structured response, include the appropriate Accept header:

Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Statusz

Example /statusz response

{
  "kind": "Statusz",
  "apiVersion": "config.k8s.io/v1alpha1",
  "metadata": {
    "name": "kube-apiserver"
  },
  "startTime": "2025-10-29T00:30:01Z",
  "uptimeSeconds": 856,
  "goVersion": "go1.23.2",
  "binaryVersion": "1.35.0",
  "emulationVersion": "1.35",
  "paths": [
    "/healthz",
    "/livez",
    "/metrics",
    "/readyz",
    "/statusz",
    "/version"
  ]
}

Example /flagz response

Use the header:

Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Flagz

{
  "kind": "Flagz",
  "apiVersion": "config.k8s.io/v1alpha1",
  "metadata": {
    "name": "kube-apiserver"
  },
  "flags": {
    "advertise-address": "192.168.8.4",
    "allow-privileged": "true",
    "authorization-mode": "[Node,RBAC]",
    "enable-priority-and-fairness": "true",
    "profiling": "true"
  }
}

Why structured responses matter

1. Automated health checks & monitoring – Tools can now extract fields directly (e.g., verify the emulation version or critical flag values) without fragile text parsing.
2. Better debugging tools – Developers can diff configurations across components, track drift over time, and build sophisticated diagnostics.
3. API versioning & stability – Introducing versioned APIs (v1alpha1) gives a clear migration path (future v1beta1, then v1) so tooling remains stable across releases.

How to use structured z‑pages

Prerequisites

Both endpoints require the corresponding feature gates to be enabled:

Example: Getting structured responses with curl

# Structured statusz response
curl \
  --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
  --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
  --cacert /etc/kubernetes/pki/ca.crt \
  -H "Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Statusz" \
  https://localhost:6443/statusz | jq .

# Structured flagz response
curl \
  --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
  --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
  --cacert /etc/kubernetes/pki/ca.crt \
  -H "Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Flagz" \
  https://localhost:6443/flagz | jq .

Note

• The examples use client‑certificate authentication and verify the server’s certificate with --cacert.
• In a test environment you can bypass verification with --insecure (or -k), but never do this in production – it exposes you to man‑in‑the‑middle attacks.

Important considerations

Alpha feature status

• The structured z‑page responses are alpha in Kubernetes 1.35.
• The API format may change in future releases.
• These endpoints are intended for debugging, not for production‑critical automation.
• Avoid relying on them for critical monitoring until they reach beta or stable status.

Security & access control

z‑pages expose internal component information, so proper access controls are essential:

• Restrict access to trusted users or service accounts.
• Use network policies, RBAC, and TLS client authentication to limit exposure.
• Regularly audit who can reach the /*z endpoints.

TL;DR

• Kubernetes 1.35 adds optional JSON output to /statusz and /flagz.
• Request JSON with the Accept: application/json;v=v1alpha1;g=config.k8s.io;as=… header.
• Feature gates ComponentStatusz and ComponentFlagz must be enabled.
• Use the structured data for automated checks, richer debugging tools, and future‑proof integrations—while remembering that the feature is still alpha and should be protected by strict access controls.

Access to z‑page Endpoints

Access to z‑page endpoints is restricted to members of the system:monitoring group, which follows the same authorization model as other debugging endpoints such as /healthz, /livez, and /readyz. This ensures that only authorized users and service accounts can access debugging information.

• If your cluster uses RBAC, manage access by granting the appropriate permissions to this group.

Authentication

The authentication requirements for these endpoints depend on your cluster’s configuration.

• Unless anonymous authentication is enabled, you typically need to use authentication mechanisms (e.g., client certificates) to access these endpoints.

Information Disclosure

These endpoints reveal configuration details about your cluster components, including:

• Component versions and build information
• All command‑line arguments and their values (confidential values are redacted)
• Available debug endpoints

Recommendation: Only grant access to trusted operators and debugging tools. Avoid exposing these endpoints to unauthorized users or automated systems that don’t require this level of access.

Future Evolution

As the feature matures, the Kubernetes SIG Instrumentation team expects to:

1. Introduce v1beta1 and eventually v1 versions of the API.
2. Gather community feedback on the response schema.
3. Potentially add additional z‑page endpoints based on user needs.

Try It Out

We encourage you to experiment with structured z‑pages in a test environment:

1. Enable the ComponentStatusz and ComponentFlagz feature gates on your control‑plane components.
2. Query the endpoints using both plain‑text and structured formats.
3. Build a simple tool or script that consumes the structured data.
4. Share your feedback with the community.

Learn More

• z‑pages documentation
• KEP‑4827: Component Statusz
• KEP‑4828: Component Flagz
• Join the discussion in the #sig-instrumentation channel on Kubernetes Slack

Get Involved

We’d love to hear your feedback! The structured z‑pages feature is designed to make Kubernetes easier to debug and monitor. Whether you’re:

• Building internal tooling
• Contributing to open‑source projects
• Exploring the feature

your input helps shape the future of Kubernetes observability.

• Questions, suggestions, or issues? Reach out to SIG Instrumentation on Slack or attend our regular community meetings.

Happy debugging!