Kyverno Policy Conflicts: Why “Last-Writer-Wins” Is a Production Bug

Source: Dev.to

▶️ Introduction

Kyverno has become the go‑to policy engine for Kubernetes because it feels simple:

• Policies are YAML
• They run at admission time
• No custom controllers required

Most Kyverno incidents are not bugs in Kyverno.
They are caused by incorrect assumptions about how policies are executed.

🤔 The Assumption Almost Everyone Makes

Most teams implicitly assume at least one of the following:

• Policies run in the order they are applied.
• Cluster‑scoped policies are evaluated before namespace‑scoped ones.
• Kyverno resolves conflicts deterministically.

⚠️ None of these assumptions are true.

Kyverno provides:

• No policy priority.
• No guaranteed ordering across policies.
• No conflict detection between mutations.

🧠 How Kyverno Actually Processes an Admission Request

For a single admission request, Kyverno evaluates policies in three distinct phases. The order of these phases is guaranteed.

What is not guaranteed

• Order between different mutate policies – Kyverno does not define which mutate policy runs first.
• Order between mutate rules that belong to different policies – The relative execution order is undefined.

Note:

• Within a single policy, rules are processed top‑to‑bottom.
• Across multiple policies, the order is explicitly undefined.

📌 Concrete Failure Modes

🧪 Example 1 – Mutate vs Mutate: Where Things Quietly Break

⚠️ The examples are written for Kyverno v1.17 and the policies are created at the same time.

Two MutatingPolicy objects match the same Pod.

Policy A

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: enforce-non-root
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsNonRoot: true
              }
            }
          }

Policy B

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: force-run-as-user-0
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsUser: 0
              }
            }
          }

What Kyverno sees

🧪 Example 2 – Mutate vs Validate: Self‑Inflicted Admission Failure

Scenario

One policy mutates a resource to enforce a default. Another policy validates the same field assuming a different expected state. Both policies are correct in isolation.

MutatingPolicy – Add default label

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: add-default-app-label
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            metadata: Object.metadata{
              labels: Object.metadata.labels{
                app: "default"
              }
            }
          }

ValidatingPolicy – Require a specific label value

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-app-frontend
spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - message: "Label app=frontend is required"
      expression: "object.metadata.?labels.app.orValue('') == 'frontend'"

What actually happens

1. Mutation runs first and sets app=default.
2. Validation runs after the mutation.
3. Validation fails because the required label app=frontend is missing.

❗ Admission denied – even though the original Pod would have satisfied the validation rule.

📉 Why This Gets Worse at Scale

When you have only one or two policies, the interactions are easy to reason about. As the number of policies grows, several problems emerge:

Common Issues at Scale

• Multiple teams mutate the same fields
• Validation rules encode different assumptions
• Scopes overlap silently

Typical Failures

• They don’t always happen immediately.
• They often appear only for specific workloads.
• They are hard to trace back to policy interaction rather than to the policy logic itself.

Why It Happens

Kyverno evaluates each policy independently. It does not infer intent or resolve conflicts between policies, which leads to the issues described above.

💡 Key Insight

Kyverno evaluates policies independently; it does not infer or reconcile intent across them. When intent is fragmented across multiple policies, Kyverno will enforce all matching rules, even if their combined effects are contradictory.

Part II presents design patterns to prevent this class of conflicts.