The Pitfalls of Test Coverage: Introducing Mutation Testing with Stryker and Cosmic Ray

Source: Dev.to

Overview

Goal: Overcome the limitations of code‑coverage metrics and introduce mutation testing to verify that test code actually catches errors in business logic.

Scope: Core modules of the enterprise orchestrator project (Ochestrator) in both Frontend (TypeScript) and Backend (Python).

Expected Results: Improve code stability and test reliability by securing a mutation score beyond simple line coverage.

We often believe that high test coverage means safe code. However, it’s difficult to answer the question:

“Who tests the tests?”

Tests that simply execute code without proper assertions still contribute to coverage metrics. To solve this coverage trap, we introduced mutation testing.

Implementation

1. TypeScript Environment – Stryker Mutator

For the TypeScript environment (frontend and common utilities) we chose Stryker. It integrates well with Vitest and is easy to configure.

Tech Stack: TypeScript, Vitest, Stryker Mutator

Key Configuration (stryker.config.json)

{
  "testRunner": "vitest",
  "reporters": ["html", "clear-text", "progress"],
  "concurrency": 4,
  "incremental": true,
  "mutate": [
    "src/utils/**/*.ts",
    "src/services/**/*.ts"
  ]
}

We enabled the incremental option to run tests only on files that have changed.

2. Python Environment – Cosmic Ray

For the backend we introduced Cosmic Ray. It generates powerful mutations by manipulating the AST (Abstract Syntax Tree) using Python’s dynamic nature.

Tech Stack: Python, Pytest, Cosmic Ray, Docker

Execution Architecture: Mutation testing is resource‑intensive, so we run it in parallel across multiple Docker workers.

# Partial docker-compose.test.yaml
cosmic-worker-1:
  command: uv run cosmic-ray worker cosmic.sqlite

cosmic-runner:
  depends_on: [cosmic-worker-1, cosmic-worker-2]
  command: |
    uv run cosmic-ray init cosmic-ray.toml cosmic.sqlite
    uv run cosmic-ray exec cosmic-ray.toml cosmic.sqlite

Debugging / Challenges

Real‑world Case: Survived Mutants in VideoSplitter.ts

VideoSplitter.ts handles video splitting. It had > 95 % line coverage, yet Stryker revealed many surviving mutants.

Problem Statement

// Original code
if (availableMemory  {
  // Simulate situations where memory is exactly equal to or slightly less than requiredMemory
  // ... reinforced test code ...
});

Results

• Discovered & removed 12 surviving mutants in core utility modules.
• Elevated test code from merely executing code to truly verifying it.

Key Metrics

Key Takeaways

• Coverage is just the beginning – line coverage tells you what is not tested, not the quality of what is tested.
• Mutation testing is expensive but worth it – runs can take tens of minutes, but the payoff is huge for core business logic.
• Incremental adoption – start with critical infrastructure code (e.g., VideoSplitter) to build success stories before expanding.

Verification Checklist

• Overview – goals and scope are clear.
• Implementation – tech stack and code examples are included.
• Debugging – at least one specific problem and its solution are described.
• Results – numerical data and performance indicators are provided.
• Key Takeaways – lessons learned and future plans are outlined.

Length Guidelines

• Overall: 400–800 lines (currently ~100 lines – can be expanded if needed).