How Multi-Agent AI Systems Use Screenshots as Shared Ground Truth

Source: Dev.to

[![Custodia-Admin](https://media2.dev.to/dynamic/image/width=50,height=50,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3781068%2F0488dc6e-2031-4a2e-b592-3153cd476dd7.png)](https://dev.to/custodiaadmin)

# How Multi‑Agent AI Systems Use Screenshots as Shared Ground Truth

**Source:** [Dev.to](https://dev.to/custodiaadmin/how-multi-agent-ai-systems-use-screenshots-as-shared-ground-truth-30f6)

You deploy three AI agents to run in parallel:

- **Agent A** checks the checkout flow.  
- **Agent B** verifies that pricing displays correctly.  
- **Agent C** audits form validation.

An hour later, they report conflicting results:

- Agent A saw a working cart.  
- Agent B saw missing prices.  
- Agent C’s form‑validation report contradicts Agent A’s observations.

**What went wrong?**  
They weren’t looking at the same page—they weren’t in sync.

This is the **coordination problem** in parallel multi‑agent systems. When agents execute browser tasks simultaneously, they diverge on visual reality. One agent sees the page in state X, another sees state Y, and they make contradictory decisions. The workflow fails.

The Root Cause: Text‑Only Coordination

Today’s multi‑agent systems coordinate using API responses and HTML parsing. For example:

• Agent A parses: “Cart total: $99”.
• Agent B parses: “Price tag not found”.
• Agent C parses: “Form field is visible”.

But they never actually saw the page—they only saw the HTML. CSS might have hidden the price, JavaScript might not have loaded, and a form field that appears in the markup could be off‑screen or behind a modal.

Result: agents work from incomplete, conflicting signals.

The Solution: Visual Ground Truth

Add a screenshot to every agent’s execution record.

• Agent A – when it calls verify checkout, it receives a screenshot proving what actually rendered.
• Agent B – when it checks pricing, it captures visual proof of the displayed price.
• Agent C – its form‑validation step includes a screenshot of the actual form state.

Now all three agents share verified visual reference points. They can see:

• “Cart was actually visible, not hidden by CSS.”
• “Price rendered on page, confirmed by screenshot.”
• “Form field was interactive, not disabled.”

With visual ground truth, agents stay synchronized and workflows succeed.

Real‑World Example: Parallel Checkout Testing

import json
import urllib.request
from concurrent.futures import ThreadPoolExecutor

import anthropic

client = anthropic.Anthropic()


# ── Tool: Screenshot verification ──────────────────────────────────────────────
def verify_checkout_step(step_name: str, url: str) -> dict:
    """
    Agent task: verify one checkout step with screenshot proof.

    Parameters
    ----------
    step_name: str
        Human‑readable name of the checkout step (e.g., "cart").
    url: str
        URL of the page to be captured.

    Returns
    -------
    dict
        Verification result containing the step name, a boolean flag,
        the screenshot image (base‑64), and a status message.
    """
    # Define the tool the model may call
    tools = [
        {
            "name": "screenshot",
            "description": "Capture visual proof of page state",
            "input_schema": {
                "type": "object",
                "properties": {
                    "url": {"type": "string"},
                    "width": {"type": "integer", "default": 1280},
                },
                "required": ["url"],
            },
        }
    ]

    # Prompt the model
    messages = [
        {
            "role": "user",
            "content": (
                f"Verify the {step_name} step of checkout. "
                "Take a screenshot and report if the page rendered correctly."
            ),
        }
    ]

    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=512,
        tools=tools,
        messages=messages,
    )

    # ── Capture screenshot if the model requested the tool ───────────────────────
    if response.stop_reason == "tool_use":
        for block in response.content:
            if block.type == "tool_use" and block.name == "screenshot":
                api_key = "YOUR_API_KEY"
                payload = json.dumps({"url": url}).encode()
                req = urllib.request.Request(
                    "https://pagebolt.dev/api/v1/screenshot",
                    data=payload,
                    headers={
                        "x-api-key": api_key,
                        "Content-Type": "application/json",
                    },
                    method="POST",
                )
                with urllib.request.urlopen(req) as resp:
                    result = json.loads(resp.read())
                    return {
                        "step": step_name,
                        "verified": True,
                        "screenshot_proof": result["image"],
                        "status": "Page rendered successfully",
                    }

    # If we get here the verification failed
    return {"step": step_name, "verified": False, "status": "Verification failed"}


# ── Run three agents in parallel ───────────────────────────────────────────────
checkout_steps = [
    ("cart", "https://example.com/checkout/cart"),
    ("shipping", "https://example.com/checkout/shipping"),
    ("payment", "https://example.com/checkout/payment"),
]

with ThreadPoolExecutor(max_workers=3) as executor:
    results = executor.map(lambda x: verify_checkout_step(*x), checkout_steps)

# ── Aggregate results with shared visual evidence ─────────────────────────────
verification_report = {
    "timestamp": "2026-03-04T15:30:00Z",
    "checkout_verification": list(results),
    "ground_truth_method": "PageBolt screenshots",
    "all_agents_synchronized": True,
}

print(json.dumps(verification_report, indent=2))

What this achieves

• Parallel execution – three agents verify different checkout steps at the same time.
• Concrete proof – each result contains a screenshot (base‑64 image) as ground‑truth evidence.
• Eliminates ambiguity – no “did the page actually load?” guesswork; the visual proof is shared.
• Synchronized reporting – a single aggregated report shows the status of every step.

Why This Matters at Scale

As multi‑agent systems become more sophisticated, coordination becomes critical.

By anchoring every decision to a concrete visual snapshot, multi‑agent systems can avoid the classic coordination problem and operate reliably at scale.

Website

Shared screenshots prove what all agents actually saw.

The PageBolt Advantage

Self‑hosted solutions (Puppeteer, Playwright) give you screenshots — but coordination is your problem. You have to manage infrastructure, syncing, storage, and retrieval.

PageBolt handles it: one API endpoint, instant visual proof, permanent audit history accessible to all agents. The screenshot is stored, indexed, and retrievable by any agent that needs verification.

• Your agents stay in sync.
• Your workflows scale reliably.

Try It Now

1. Get your API key at pagebolt.dev (free tier: 100 requests/month).
2. Add the screenshot tool to your multi‑agent system.
3. Deploy agents in parallel with confidence.

They’ll all see the same verified visual reality.

Your workflows will actually coordinate.