Agentic CI with Buildkite: Three practical examples

Source: Dev.to

If you read our previous post about how AI is reshaping CI, you already know about the new building blocks we added to the Buildkite platform. We call these agentic workflow components—composable primitives that give platform teams the tools they need to bring AI‑assisted processes into their CI/CD workflows.

In this post we show three practical examples that use AI agents to solve common real‑world problems:

• Reviewing GitHub pull requests
• Automatically fixing broken PR builds
• Generating first‑draft PRs from Linear issues

Each example is backed by a fully functioning GitHub repository containing a Buildkite pipeline template that you can fork, set up, run, and adapt to your team’s needs.

---

Recap of the workflow components

• Buildkite MCP server – gives agents fine‑grained access to the Buildkite REST API through specialized MCP tools.
• Buildkite model providers – connect to frontier models (e.g., Anthropic’s Claude Code) using your own API credentials or a hosted Buildkite‑managed key.
• Buildkite pipeline triggers – inbound webhooks that invoke a Buildkite pipeline in response to any external event (first‑class support for GitHub, Linear, etc.).
• Buildkite SDK – compose and generate pipeline definitions dynamically at runtime using JavaScript, TypeScript, Python, Go, Ruby, and other languages.
• AI‑powered plugins – powered by Claude, Codex, Amazon Bedrock, and others, making it easy to annotate CI jobs with rich build summaries.

Together these building blocks let you build flexible, adaptive workflows that bring AI agents into your CI/CD process on your own terms.

Example 1: GitHub code‑review bot

Code review is essential but time‑consuming, especially as AI agents generate more code for humans to review. An AI‑powered reviewer can provide a first‑pass review, surface non‑obvious bugs, and save valuable time.

The example repository is available on GitHub at buildkite-agentic-examples/github-code-review-bot.

How it works

1. A label is applied to a PR, triggering a Buildkite pipeline via a pipeline trigger (an inbound webhook URL tied to a specific pipeline).
2. The pipeline runs a script that parses and validates the webhook payload.
3. The script appends a step to the running Buildkite pipeline to spawn an AI agent.
4. The agent completes the review task and posts a comment on the PR.

All three examples in this post follow this general pattern.

Handler script (scripts/handler.ts)

The core logic lives in a Node.js program written in TypeScript. It uses the Buildkite and GitHub SDKs to run Claude Code with a task‑specific prompt.

import { execSync } from "child_process";
import { Pipeline } from "@buildkite/buildkite-sdk";
import { Octokit } from "octokit";

// ...

// Generate the pipeline with the Buildkite SDK.
function generateCodeReviewPipeline(
  webhookPullRequestUrl: string,
  agentBuildUrl: string
): string {
  const pipeline = new Pipeline();
  const tokenArgs = [
    `PullRequestURL=${webhookPullRequestUrl}`,
    `AgentBuildURL=${agentBuildUrl}`,
  ];

  pipeline.addStep({
    label: ":buildkite: Reviewing the code",
    commands: [...runAgent(tokenArgs)],
    plugins: {
      docker: {
        image: "buildkite-agentic-example-tools:latest",
        "mount-checkout": false,
        "mount-buildkite-agent": true,
        environment: [
          // ...
          "TRIGGER_ON_LABEL",
          "MODEL_PROVIDER",
        ],
      },
    },
    // ...
  });

  return pipeline.toYAML();
}

async function main() {
  // Fetch the incoming payload from Buildkite.
  const event = buildkiteAgent("meta-data", "get", "buildkite:webhook").trim();
  const payload = JSON.parse(event);

  // ...

  // Exit unless the payload has a label matching the one we're listening for.
  const labelName = payload.label.name;
  if (labelName !== process.env.TRIGGER_ON_LABEL) {
    console.log(`Label is not '${process.env.TRIGGER_ON_LABEL}', exiting`);
    process.exit(0);
  }

  // ...

  // Generate and upload a new pipeline step to run the AI agent.
  const pipelineYaml = generateCodeReviewPipeline(pullRequestUrl);
  execSync("buildkite-agent pipeline upload", {
    input: pipelineYaml,
    encoding: "utf-8",
  });
}

main().catch((error) => {
  console.error("Error:", error.message);
  process.exit(1);
});

The step runs Claude in a Docker container (via the Buildkite Docker plugin) that contains only the tools Claude needs for this task: Node.js, the GitHub CLI, the local Buildkite MCP server, the Claude Code CLI, and the required scripts, prompts, and environment variables. Running the agent in a container adds isolation and safety, though it isn’t strictly required.

Inside the container, Claude:

• Clones the PR’s repository
• Checks out the PR branch
• Analyzes the changes
• Posts a review comment on the PR
• Annotates the Buildkite build using the MCP server’s annotation tooling

Pipeline configuration (.buildkite/pipeline.yml)

All settings—including the model provider and the trigger label—are configurable in the pipeline YAML:

secrets:
  GITHUB_TOKEN: GITHUB_TOKEN
  BUILDKITE_API_TOKEN: API_TOKEN_BUILDKITE

env:
  GITHUB_CLI_VERSION: "2.83.0"
  BUILDKITE_MCP_SERVER_VERSION: "0.7.3"
  TRIGGER_ON_LABEL: "buildkite-review"
  MODEL_PROVIDER: "anthropic"

steps:
  - label: ":node: Generate the pipeline"
    command: |
      # Generate and upload the pipeline to handle the webhook.
      echo "--- :webhook: Run the webhook handler"
      npm install && npm run build
      node dist/handler

Claude CLI configuration (scripts/claude.sh)

The Claude CLI does not communicate directly with the Anthropic API. Instead, it uses a Buildkite‑managed model‑provider endpoint that proxies the Anthropic API. The script sets the required environment variables from Buildkite runtime values:

#!/bin/bash
# ...

# Set up Buildkite Hosted Models
export ANTHROPIC_BASE_URL="$BUILDKITE_AGENT_ENDPOINT/ai/anthropic"
export ANTHROPIC_API_KEY="$BUILDKITE_AGENT_ACCESS_TOKEN"

# ...

echo "--- :robot_face: Starting Claude Code"
echo "$prompt" | claude -p --mcp-config mcp.json

These values are injected automatically by Buildkite at runtime, giving you a seamless way to use Claude—or any supported model provider—without exposing your own credentials. See the model providers documentation for further details.