I gave myself 48 hours to validate an AI code review tool. Here's how I built it.
Source: Dev.to
Idea
I wanted to test a simple question: what if every pull request you opened got an instant AI code review?
Not a replacement for human reviewers — just a first pass that catches the obvious issues before your teammates even look at it.
Build in 48 Hours
I gave myself 48 hours to build it, ship it, and see if anyone actually finds it useful.
How It Works
CodeReview.ai is a GitHub App. After installing it on a repository, every time a PR is opened it:
- Receives a webhook from GitHub.
- Pulls the diff via the GitHub API.
- Sends the diff to GPT‑3.5‑turbo with a code‑review prompt.
- Posts the review as a PR comment within seconds.
That’s it—no dashboard, no config files, no pricing tiers. Install and go.
Tech Stack
- TypeScript – end‑to‑end code.
- Vercel serverless functions – webhook handler.
- Octokit – GitHub API calls.
- OpenAI API (GPT‑3.5‑turbo) – generates the review.
- PostHog – tracks installs and usage.
- Static HTML + Tailwind – landing page.
The whole system consists of two API routes: one for the GitHub webhook, one for waitlist sign‑ups.
Success Metrics
I set four metrics to decide whether to continue:
| Metric | Goal |
|---|---|
| Installs in 48 h | > 15 |
| Repos receiving ≥2 reviews | ≥ 5 |
| Waitlist signup rate | > 10 % of installs |
| Sentiment | Not negative across feedback channels |
If I hit 3 of 4, I keep building; otherwise I scrap it.
Limitations
- Diffs are truncated to 8,000 characters; large PRs won’t get full coverage.
- GPT‑3.5‑turbo is fast and cheap but not as sharp as GPT‑4; review depth has a ceiling.
- 50 reviews per day per installation to keep my OpenAI bill sane (I’m a solo dev).
Try It Out
- Landing page:
- Install the GitHub App:
If you install it and find it annoying rather than helpful, I genuinely want to hear that. The whole point of the sprint is to find out.