Why green CI doesn't mean your system works

Source: Dev.to

A case study: how a TypeScript migration doubled my test runs — with zero failures

CI was green. Tests were passing. PRs were merging.
The system was broken. Nothing in the logs showed it.

After migrating my test project from JavaScript to TypeScript, CI started taking almost twice as long to run. No failures. No errors. Just… slower. I assumed it was normal TypeScript compilation overhead and moved on—by accident.

The symptom

Near the end of the migration, when I started deleting the original .js files, the test count dropped by almost half:

• Before: ~240 tests
• After: ~120 tests

That number didn’t make sense. I hadn’t removed any tests; I only deleted old JavaScript files that were supposed to be gone already.

The cause

Playwright was picking up both .spec.js and .spec.ts files at the same time. Every test in the suite was running twice—same assertions, same setup, same teardown—duplicated silently, without a single warning.

The worst part wasn’t the wasted time. CI made it look like things were improving. Runtime crept up gradually, which read as “normal post‑migration slowdown.” I had a plausible story for the symptom, so I stopped looking.

playwright.config.ts had no explicit testMatch. Playwright’s default glob matches both .js and .ts files, so it picked up everything.

The fix

// playwright.config.ts
export default defineConfig({
  // …
  testMatch: ['**/*.spec.ts'],
});

Adding the testMatch line stopped the duplicate execution.

Lessons learned

• CI validates execution, not correctness. Green CI only means nothing crashed during execution; it doesn’t guarantee the right tests ran, in the right quantity, with the right assumptions.

• A simple discovered‑tests counter in CI could have caught the deviation. Now the pipeline fails explicitly if the test count differs from the expected value.

• Most problems in test systems don’t show up as failures. They appear as:

  • duplicated execution
  • silent performance degradation
  • runner behaviour changes with no test changes

  And none of them have alerts because we don’t design for them.

Failure signature

• CI green
• Runtime doubled
• Test count doubled
• Zero warnings

The hidden assumption was “I assumed a slower CI run meant normal post‑migration overhead.” In reality, the runner had been doing twice the work for weeks—silently, without a single warning.

Part of the Silent Failures in Test Automation series

Full project (API + UI + E2E + CI + AI endpoint): GitHub