Don't Feed HTML to Your Agents

Source: Dev.to

TL;DR — Give AI a White Box, Not a Black Box

Most AI agents interact with web apps through Black Box methods: consuming DOM dumps or screenshots, then guessing what to click. HTML was never designed for machines. From the AI’s perspective, the DOM is noise where business logic is faintly buried.

This essay argues for a White Box approach: expose a Semantic State Layer that reveals the application’s structure, rules, state, and valid transitions directly. This is not about replacing the UI; it’s about providing an Intelligence Interface (II) alongside the traditional User Interface.

Try it yourself → Manifesto Playground

1. Black Box: The Current State of AI + Web Apps

Here’s how most teams “add AI” to their web apps today:

• Use LangChain, AutoGPT, or browser automation
• Drive Playwright or Puppeteer
• Dump the DOM or screenshot into the model
• Hope it figures out what to click

This is the Black Box approach. The agent sees only the rendered surface and must infer everything else.

What’s Wrong with DOM Dumps?

<div class="product-name">
  <label>Product Name</label>
  <input type="text" name="name" />
  <span class="error">This field is required.</span>
</div>

From an agent’s perspective:

The agent is forced to guess. A CSS refactor breaks everything. A layout change confuses the model. The logic was never exposed — only its visual projection.

Signal 90 %.

2. White Box: Exposing the Application’s Brain

The alternative is a White Box protocol. Instead of showing HTML, the engine exposes a Semantic Snapshot — a structured representation of the application’s internal state that agents can read directly.

{
  "topology": {
    "viewId": "product-create",
    "mode": "create",
    "sections": [
      { "id": "basic", "title": "Basic Info", "fields": ["name", "productType"] },
      { "id": "shipping", "title": "Shipping", "fields": ["shippingWeight"] }
    ]
  },
  "state": {
    "form": { "isValid": false, "isDirty": false },
    "fields": {
      "name": {
        "value": "",
        "meta": { "valid": false, "hidden": false, "disabled": false, "errors": ["Required"] }
      },
      "productType": {
        "value": "PHYSICAL",
        "meta": { "valid": true, "hidden": false, "disabled": false, "errors": [] }
      },
      "shippingWeight": {
        "value": null,
        "meta": { "valid": true, "hidden": false, "disabled": false, "errors": [] }
      }
    }
  },
  "constraints": {
    "name": { "required": true, "minLength": 2, "maxLength": 100 },
    "shippingWeight": { "min": 0, "max": 2000, "dependsOn": ["productType"] }
  },
  "interactions": [
    { "id": "updateField:name", "intent": "updateField", "target": "name", "available": true },
    { "id": "updateField:productType", "intent": "updateField", "target": "productType", "available": true },
    { "id": "submit", "intent": "submit", "available": false, "reason": "Name is required" }
  ]
}

Now the agent has:

• Topology – screen structure, sections, field hierarchy
• State – current values, validity, visibility, errors per field
• Constraints – required, min/max, dependencies
• Interactions – what actions are available and why some are blocked

No guessing. No inference. The agent reads the application’s brain directly.

3. A Real Use Case: “Where Do I Select the Week?”

🎮 See it in action: Manifesto Playground – try changing field values and watch the semantic state update in real‑time.

Scenario

User: “I see a date picker, but where do I select which week?”

AI Chatbot: “The week selector only appears when you set frequency to ‘Weekly’. Right now it’s set to ‘Daily’. Should I change it for you?”

For this to work, the AI needs to know:

• A field called weekSelector exists and is currently hidden
• It becomes visible when frequency === 'WEEKLY'
• The current value of frequency is 'DAILY'

A DOM‑only approach cannot provide this reliably, but a Semantic Snapshot can:

{
  "fields": {
    "frequency": {
      "value": "DAILY",
      "meta": { "hidden": false }
    },
    "weekSelector": {
      "value": null,
      "meta": { "hidden": true },
      "visibleWhen": "frequency === 'WEEKLY'"
    }
  }
}

The AI reads this and knows—without inference—exactly why the field is hidden and what would make it appear.

4. The Protocol Loop

Manifesto implements a continuous feedback loop between the engine and AI agents:

┌─────────────────────────────────────────────────────────────────────┐
│                                                                     │
│  [Context Injection] → [Reasoning] → [Action Dispatch] → [Delta]    │
│          ▲                                                  │       │
│          └─────────────── Continuous Snapshots ─────────────┘       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Step‑by‑step

1. Context Injection – Engine exports a Semantic Snapshot containing topology, state, constraints, and interactions.
2. Reasoning – Agent plans the next action based on the snapshot.
3. Action Dispatch – Agent calls abstract intents (e.g., updateField, submit, reset, validate) instead of raw DOM events.
4. Delta Feedback – Engine returns what changed, not just “success”. The diff shows causality (e.g., “I changed X, and Y became hidden”).
5. The loop repeats with an updated snapshot, giving the agent predictable, structured feedback rather than “click and hope”.

5. The API: Exploration and Execution

Manifesto exposes this protocol through @manifesto-io/ai.

Exploration Mode – “What can I do here?”

import { createInteroperabilitySession } from '@manifesto-io/ai'

const session = createInteroperabilitySession({
  runtime,       // FormRuntime instance
  viewSchema,    // View definition
  entitySchema,  // Entity definition
})

// Get the current semantic snapshot
const snapshot = session.snapshot()

// snapshot.interactions tells the agent:
// - submit: available = false, reason = "Name is required"

From here, the agent can query available interactions, update fields, submit forms, and receive incremental diffs, all driven by the white‑box semantic representation.