The Problem with AI Tests That Don't Know Your App

Source: Dev.to

AI‑generated Cypress tests

AI-generated Cypress tests are promising — but by default, the AI has never seen your app.

The interesting part isn’t “look, the AI wrote a test.” The interesting part is whether an AI grounded in your team’s own Swagger spec, component docs, and bug history can cover things you would miss.

Retrieval‑Augmented Generation (RAG)

That’s where RAG comes in. RAG (Retrieval‑Augmented Generation) is the pattern of feeding your own documents to an AI at query time. Instead of a generic model guessing at your button labels and API routes, it works from the same source of truth your team already uses.

Cypress’s experimental cy.prompt() command

Pair that with cy.prompt() — Cypress’s experimental AI‑native test authoring command — and something interesting happens. The AI works with more precision. It can map to your endpoints. It may even surface flows you forgot to cover.

Limitations

That said, it’s not a silver bullet. The human still writes better assertions. The AI covers breadth, the human covers intent. And any context that never made it into your docs won’t make it into your tests either.

Open question

If you’ve tried AI‑generated tests for your app: how much did the AI actually know about it?