The Pitfalls of Test Coverage: Introducing Mutation Testing with Stryker and Cosmic Ray

Source: Dev.to

[![wintrover](https://media2.dev.to/dynamic/image/width=50,height=50,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1896056%2Fa00c0da2-64c2-47ab-b5d0-2256c4ba0e75.png)](https://dev.to/wintrover)

# Overview

**Goal**: Overcome the limitations of code‑coverage metrics and introduce *mutation testing* to verify that test code actually catches errors in business logic.

**Scope**: Core modules of the enterprise orchestrator project (**Ochestrator**) in both Frontend (TypeScript) and Backend (Python).

**Expected Results**: Improve code stability and test reliability by securing a *mutation score* beyond simple line coverage.

We often believe that high test coverage means safe code. However, it’s difficult to answer the question:

> **“Who tests the tests?”**

Tests that simply execute code without proper assertions still contribute to coverage metrics. To solve this *coverage trap*, we introduced mutation testing.

![Mutation Testing Flow](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fb9d8oqhy3i2k3o2phh1z.png)

---

## Implementation

### 1. TypeScript Environment – Stryker Mutator

For the TypeScript environment (frontend and common utilities) we chose **[Stryker](https://stryker-mutator.io/)**. It integrates well with Vitest and is easy to configure.

**Tech Stack**: TypeScript, Vitest, Stryker Mutator  

**Key Configuration (`stryker.config.json`)**

```json
{
  "testRunner": "vitest",
  "reporters": ["html", "clear-text", "progress"],
  "concurrency": 4,
  "incremental": true,
  "mutate": [
    "src/utils/**/*.ts",
    "src/services/**/*.ts"
  ]
}

We enabled the incremental option to run tests only on files that have changed.

---

2. Python Environment – Cosmic Ray

For the backend we introduced Cosmic Ray. It generates powerful mutations by manipulating the AST (Abstract Syntax Tree) using Python’s dynamic nature.

Tech Stack: Python, Pytest, Cosmic Ray, Docker

Execution Architecture: Mutation testing is resource‑intensive, so we run it in parallel across multiple Docker workers.

# Partial docker-compose.test.yaml
cosmic-worker-1:
  command: uv run cosmic-ray worker cosmic.sqlite

cosmic-runner:
  depends_on: [cosmic-worker-1, cosmic-worker-2]
  command: |
    uv run cosmic-ray init cosmic-ray.toml cosmic.sqlite
    uv run cosmic-ray exec cosmic-ray.toml cosmic.sqlite

---

Debugging / Challenges

Real‑world Case: Survived Mutants in VideoSplitter.ts

VideoSplitter.ts handles video splitting. It had > 95 % line coverage, yet Stryker revealed many surviving mutants.

Problem Statement

// Original code
if (availableMemory  {
  // Simulate situations where memory is exactly equal to or slightly less than requiredMemory
  // ... reinforced test code ...
});

---

Results

• Discovered & removed 12 surviving mutants in core utility modules.
• Elevated test code from merely executing code to truly verifying it.

Key Metrics

---

Key Takeaways

• Coverage is just the beginning – line coverage tells you what is not tested, not the quality of what is tested.
• Mutation testing is expensive but worth it – runs can take tens of minutes, but the payoff is huge for core business logic.
• Incremental adoption – start with critical infrastructure code (e.g., VideoSplitter) to build success stories before expanding.

---

Verification Checklist

• Overview – goals and scope are clear.
• Implementation – tech stack and code examples are included.
• Debugging – at least one specific problem and its solution are described.
• Results – numerical data and performance indicators are provided.
• Key Takeaways – lessons learned and future plans are outlined.

---

Length Guidelines

• Overall: 400–800 lines (currently ~100 lines – can be expanded if needed).

I’m happy to help clean up your markdown! Could you please paste the markdown segment you’d like me to tidy up? Once I have the content, I’ll preserve its structure and meaning while improving formatting, consistency, and readability.