Kubernetes v1.36: Mutable Pod Resources for Suspended Jobs (beta)

Source: Kubernetes Blog

Why mutable pod resources for suspended Jobs?

Batch and machine learning workloads often have resource requirements that are not precisely known at Job creation time. The optimal resource allocation depends on current cluster capacity, queue priorities, and the availability of specialized hardware like GPUs.

Before this feature, resource requirements in a Job’s pod template were immutable once set. If a queue controller (e.g., Kueue) determined that a suspended Job should run with different resources, the only option was to delete and recreate the Job, losing any associated metadata, status, or history. This feature also allows a specific Job instance for a CronJob to progress slowly with reduced resources rather than failing outright when the cluster is heavily loaded.

Example: Updating resources for a suspended Job

Original Job (requesting 4 GPUs)

apiVersion: batch/v1
kind: Job
metadata:
  name: training-job-example-abcd123
  labels:
    app.kubernetes.io/name: trainer
spec:
  suspend: true
  template:
    metadata:
      annotations:
        kubernetes.io/description: "ML training, ID abcd123"
    spec:
      containers:
      - name: trainer
        image: example-registry.example.com/training:2026-04-23T150405.678
        resources:
          requests:
            cpu: "8"
            memory: "32Gi"
            example-hardware-vendor.com/gpu: "4"
          limits:
            cpu: "8"
            memory: "32Gi"
            example-hardware-vendor.com/gpu: "4"
      restartPolicy: Never

Updated Job (only 2 GPUs available)

apiVersion: batch/v1
kind: Job
metadata:
  name: training-job-example-abcd123
  labels:
    app.kubernetes.io/name: trainer
spec:
  suspend: true
  template:
    metadata:
      annotations:
        kubernetes.io/description: "ML training, ID abcd123"
    spec:
      containers:
      - name: trainer
        image: example-registry.example.com/training:2026-04-23T150405.678
        resources:
          requests:
            cpu: "4"
            memory: "16Gi"
            example-hardware-vendor.com/gpu: "2"
          limits:
            cpu: "4"
            memory: "16Gi"
            example-hardware-vendor.com/gpu: "2"
      restartPolicy: Never

After updating the resources, the controller resumes the Job by setting spec.suspend to false. The new Pods are created with the adjusted resource specifications.

How it works

The Kubernetes API server relaxes the immutability constraint on pod‑template resource fields only for suspended Jobs. No new API types are introduced; the existing Job and pod‑template structures accommodate the change through relaxed validation.

Mutable fields

• spec.template.spec.containers[*].resources.requests
• spec.template.spec.containers[*].resources.limits
• spec.template.spec.initContainers[*].resources.requests
• spec.template.spec.initContainers[*].resources.limits

Conditions for resource updates

1. The Job has spec.suspend: true.
2. For a Job that was previously running and then suspended, all active Pods must have terminated (status.active == 0) before resource mutations are accepted.
3. Standard resource validation still applies (e.g., limits ≥ requests, extended resources as whole numbers where required).

What’s new in beta

• The MutablePodResourcesForSuspendedJobs feature gate is enabled by default in Kubernetes v1.36.
• Clusters running v1.36 can use the feature without any additional API‑server configuration.

Try it out

If your cluster runs Kubernetes v1.36 or later, the feature is available out of the box. For v1.35 clusters, enable the feature gate on the kube‑apiserver.

# Create a suspended Job
kubectl apply -f my-job.yaml --server-side

# Edit the resource requests
kubectl edit job training-job-example-abcd123

# Resume the Job
kubectl patch job training-job-example-abcd123 -p '{"spec":{"suspend":false}}'

Considerations

Running Jobs that are suspended

When suspending a Job that was already running, you must wait for all active Pods to terminate before modifying resources. The API server rejects mutations while status.active > 0 to prevent inconsistencies between running Pods and the updated pod template.

Pod replacement policy

If your Job may have failed Pods, consider setting:

podReplacementPolicy: Failed

This ensures replacement Pods are created only after previous Pods have fully terminated, avoiding resource contention from overlapping Pods.

ResourceClaims

Dynamic Resource Allocation (DRA) resourceClaimTemplates remain immutable. Workloads that use DRA must recreate the claim templates separately to match any resource changes.

Getting involved

The feature was developed by SIG Apps with input from WG Batch. Feedback is welcome as the feature progresses toward stable.

• Slack: #sig-apps
• Slack: #wg-batch
• KEP tracking issue: KEP‑5440