Stop Shipping 'Zombie Tests': Introducing Project Vandal v0.2.0

Source: Dev.to

What You’ll Learn

• The Zombie Test Problem – Why passing tests can be more dangerous than failing ones
• Runtime UI Mutation – How Vandal sabotages the live DOM instead of rebuilding source code
• Shadow DOM Support – Penetrate modern web components that hide from standard selectors
• Kill Ratio Metrics – Quantify your test suite’s actual resilience
• Quick Integration – Drop‑in Playwright wrapper with zero test rewrites

The Zombie Test Problem

Have you ever looked at a 100 % green test suite and wondered: “Is this actually testing anything, or is it just passing because the happy path hasn’t changed?”
In test automation we often suffer from test rot—tests that remain green even when the application logic is broken. These are Zombie Tests.

The Hidden Danger

Zombie Tests give you a false sense of security. They are the reason bugs slip into production despite a massive automation suite.

What Is Project Vandal?

Vandal is a deterministic chaos‑engineering tool for front‑ends. Unlike traditional mutation testing that modifies source code (slow and rebuild‑heavy), Vandal sabotages the live DOM inside your browser during test execution.

The Moment

Traditional tools change your if statements to else in React/Vue source. Vandal changes the browser’s reality. It strips click listeners, shifts UI elements, and sabotages form state while the test is running.

Vandal v0.2.0: What’s New?

We’ve packed the v0.2.0 release with enterprise‑grade features designed for high‑scale apps.

1. Persistent Chaos (Navigation Survival) ⚓
   UI mutation survives page reloads and transitions using a combination of add_init_script and a deep MutationObserver.

2. Recursive Shadow DOM Support 🕵️‍♂️
   Vandal now recursively penetrates Shadow DOM boundaries, allowing elements hidden inside multiple shadow roots to be targeted.

3. Automatic Revert (Live Healing) 🩹
   Cache the original state of elements and restore them on‑the‑fly with await v.revert_mutation().

The “Vandalism” Playbook

High‑impact strategies designed to mimic real‑world regressions:

• Stealth Disable – Sets pointer-events: none. The button looks perfect, but it’s “dead” to user interaction.
• UI Shift – Translates elements by 100 px. Tests that rely on hard‑coded coordinates will fail.
• Slow Load – Simulates a 5‑second UI hang by temporarily hiding elements. Checks whether your test waits properly.
• Data Sabotage – Replaces critical labels and input values with junk data to verify data‑validation logic.

Installation

pip install project-vandal

Basic Usage

Integrate Vandal into your existing Playwright tests as an async context manager:

from vandal import Vandal

async def test_critical_path(page):
    async with Vandal(page) as v:
        # 1. Apply a persistent mutation
        await v.apply_mutation("stealth_disable", "#checkout-btn")

        # 2. Navigate – the mutation survives!
        await page.goto("https://myapp.com/cart")

        # 3. This SHOULD fail if your test is resilient
        try:
            await page.click("#checkout-btn", timeout=2000)
            print("🧟 MUTANT SURVIVED: Test is a Zombie!")
        except:
            print("💀 MUTANT KILLED: Test is Robust.")

    # Generate a beautiful HTML report
    v.save_report("ci_resilience_report.html")

Reporting: From Console to HTML

Vandal v0.2.0 now exports structured JSON and polished HTML reports. No more digging through console logs—visual scorecards are ready for your CI/CD dashboard.

High‑Impact Use Cases

• CI/CD Gatekeeping – Fail builds where more than 10 % of UI mutants survive.
• Shadow DOM Validation – Test elusive Web Components with confidence.
• Assertion Benchmarking – Quantify the “Kill Ratio” of your automation suite.

Join the Vandalism Movement

Stop counting lines of code coverage. Start measuring assertion effectiveness. Project Vandal makes “green checkmarks” mean something again.

Open Source

Project Vandal is an open‑source initiative. Check it out on PyPI and start validating your test suite’s resilience today.