Build your own AI code review agent in CI

Source: Dev.to

The problem with manual PR reviews

If you do PR reviews all day, you already know the pain:

• Reviewers miss stuff when they’re tired.
• The same comments repeat forever (naming, tests, edge cases).
• A “quick PR” turns into a 45‑minute context rebuild.

The AI‑powered solution

Imagine every pull request getting an instant, structured code review (correctness, security, performance, tests) posted automatically by CI using your choice of model (OpenAI, Anthropic, OpenRouter, or a local Ollama instance) — without paying for another “AI code review” subscription.

The key isn’t the model; it’s the review rubric (the prompt/workflow) that forces useful structure:

• Separate high‑risk issues from nitpicks.
• Demand concrete fixes and test suggestions.
• Require “what I looked at” and “what I’m unsure about”.

Why CI‑based reviews make sense

• Chat subscriptions are great for interactive use, but CI code review follows a different pricing pattern: it only runs when PRs happen.
• You can use pay‑as‑you‑go APIs, route cheaper/faster models for small diffs, or run local models (Ollama) where the marginal cost is near zero.
• With a DIY pipeline you control model choice, max tokens, when it runs, and what counts as “worth reviewing”.

What you’ll end up with

• A GitHub Action that runs on every PR.
• A code‑review workflow that outputs a structured Markdown review.

GitHub Action definition

Create the file .github/workflows/ai-code-review.yml in your repository:

name: AI Code Review
on:
  pull_request:

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install Jazz
        run: npm install -g jazz-ai

      - name: Run code review workflow
        run: jazz --output raw workflow run code-review --auto-approve
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Notes

• --output raw is convenient in CI (easy to capture/redirect).
• --auto-approve makes the step fully unattended.
• Permissions are intentionally minimal.

If you prefer a provider other than OpenAI, swap the environment variable for your provider (Anthropic, OpenRouter, etc.).

Review rubric (prompt) definition

Create the file workflows/code-review/WORKFLOW.md. This is the prompt your agent will run:

---
name: code-review
description: Review PR diff and produce a structured report
autoApprove: read-only
---

Review the current PR diff.

Output GitHub‑flavored Markdown with:

1. **Summary** (2–4 bullets)  
2. **High‑risk issues** (correctness + security)  
3. **Performance / complexity concerns**  
4. **API / UX footguns**  
5. **Test gaps + concrete test suggestions**  
6. **Nitpicks** (style/readability)

**Rules**  
- Be specific: reference files/functions.  
- Prefer minimal diffs / smallest safe fix.  
- If you’re unsure, say so and propose how to verify.  
- No generic advice (“add tests”) — propose exact test cases.  
- Rank issues (High/Medium/Low).  
- List files reviewed, assumptions, and what was not checked.

Capturing and posting the review

Add steps to the GitHub Action to write the review to a file and comment on the PR:

- name: Generate review markdown
  run: jazz --output raw workflow run code-review --auto-approve > review.md
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Comment on PR
  run: gh pr comment "$PR_NUMBER" --body-file review.md
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    PR_NUMBER: ${{ github.event.pull_request.number }}

Inline annotations are possible later, but they’re not required to get immediate value.

Best‑practice checklist

• Read‑only mode: Keep autoApprove at read‑only for review jobs; don’t let the agent mutate the repo.
• Issue ranking: Force the agent to rank issues (High/Medium/Low). If everything is “important”, nothing is.
• False‑positive budget: If the review is noisy for a week, developers will ignore it. Adjust the rubric accordingly.
• Model routing: Use a cheap model for small PRs and a stronger model for large refactors.
• Transparency: Require the agent to list files reviewed, assumptions made, and what it didn’t check.

Real‑world example

The Jazz repository uses Jazz for its own code reviews and release notes. See the workflow files here: https://github.com/lvndry/jazz/tree/main/.github

---

Thanks for reading!