How to Fix Karpenter Migration Issues During Upgrade (v0.25.0 v1.5.0)

Source: Dev.to

Introduction

This guide covers the real issues encountered while upgrading Karpenter from v0.25.0 to v1.5.0 in production, why they happened, and the exact fixes. If you’re planning this upgrade, the steps below will save you hours of debugging.

Upgrade Overview

Skipping the CRD migration step leads to controller crashes, stuck resources, and broken uninstalls.

Common Errors & Fixes

1. Helm chart not found

helm upgrade karpenter karpenter/karpenter --version 1.5.0
# Error: chart "karpenter" matching 1.5.0 not found

Why: The old Helm repository only contains versions up to 0.16.3. Karpenter v1.x was moved to an OCI registry.

Fix:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.5.0 \
  --namespace karpenter \
  --wait

2. OCI tag with v prefix

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter --version v1.5.0
# Error: ... v1.5.0: not found

Why: Starting in v0.35.0, OCI tags no longer use the v prefix.

Fix: Use the version without the v prefix.

--version 1.5.0   # NOT v1.5.0

3. Missing v1 CRDs

# ERROR: no matches for kind "NodeClaim" in version "karpenter.sh/v1"
# panic: unable to retrieve the complete list of server APIs

Why: The v1.5.0 controller requires the new v1 CRDs (NodePool, NodeClaim, EC2NodeClass). The cluster still has only the old v1alpha5 CRDs.

Fix: Install the new CRDs before upgrading the controller.

helm upgrade --install karpenter-crd \
  oci://public.ecr.aws/karpenter/karpenter-crd \
  --version 1.5.0 \
  --namespace karpenter \
  --create-namespace

Then upgrade the controller:

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.5.0 \
  --namespace karpenter \
  --wait

4. IAM permission error

"error": "not authorized to perform: ec2:DescribeImages"

Why: Karpenter v1 introduces new instance‑profile and AMI‑discovery workflows.

Fix: Add the following permissions to the Karpenter controller IAM role:

{
  "Effect": "Allow",
  "Action": [
    "ec2:DescribeImages",
    "iam:GetInstanceProfile",
    "iam:CreateInstanceProfile",
    "iam:DeleteInstanceProfile",
    "iam:AddRoleToInstanceProfile",
    "iam:RemoveRoleFromInstanceProfile"
  ],
  "Resource": "*"
}

Restart the deployment:

kubectl rollout restart deployment karpenter -n karpenter

Migration Steps

Backup Existing Resources

kubectl get provisioners -A -o yaml > provisioners-backup.yaml
kubectl get awsnodetemplates -A -o yaml > awsnodetemplates-backup.yaml

Install New CRDs (if not already done)

helm upgrade --install karpenter-crd \
  oci://public.ecr.aws/karpenter/karpenter-crd \
  --version 1.5.0 \
  --namespace karpenter

Upgrade the Controller (after CRDs)

helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version 1.5.0 \
  --namespace karpenter \
  --wait

Convert Provisioner → NodePool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized

Convert AWSNodeTemplate → EC2NodeClass

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: my-cluster
  role: "karpenter-node-role-name"

Apply the new resources:

kubectl apply -f ec2nodeclass.yaml
kubectl apply -f nodepool.yaml

Verify:

kubectl get ec2nodeclass
kubectl get nodepools

Test Provisioning

kubectl run test --image=nginx --requests=cpu=1,memory=1Gi

Drain and Clean Up Old Resources

kubectl delete provisioner default
kubectl delete awsnodetemplate default

Result

• Karpenter v1.5.0 running smoothly
• All nodes migrated to NodePools
• Cluster ready for EKS 1.32
• Zero downtime during the upgrade

References

• Official Upgrade Guide
• v1 Migration Guide

Monitoring Logs During Upgrade

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -f