# Runtime Snapshots #7 — Inside SiFR: The Schema That Makes LLMs See Web UIs

Source: Dev.to

SiFR stands for Semantic Information for Representation – a JSON schema that captures the runtime state of a web page in a way that is:

• Token‑efficient (often 10–50× smaller than raw HTML on complex pages)
• Semantically structured (models can reason over it without reconstructing the UI from markup)
• Visually aware (preserves layout relationships without pixels)

It is a preprocessing layer that sits between the DOM and your AI, turning “what the browser rendered” into “what the model can reason about”.

---

Why raw HTML isn’t enough

Typical e‑commerce pages contain:

• Deeply nested layout wrappers
• Duplicated markup for responsive layouts
• Client‑side frameworks with non‑semantic containers
• Hidden / disabled / off‑screen elements that still exist in the DOM

Sending this HTML to an LLM forces it to:

1. Reconstruct the runtime UI state
2. Solve the downstream task

Most failures happen during step 1. A “find the button” path in raw markup looks like:

div > div > div > div > div > div > ... > button

---

SiFR in practice

A simple element snapshot

{
  "id": "btn042",
  "text": "Add to Cart",
  "actions": ["clickable"],
  "salience": "high",
  "cluster": "product-actions"
}

The LLM sees what the element is, how it behaves, and where it belongs—without reverse‑engineering UI meaning from markup.

Full SiFR snapshot structure

1. Page‑level context – URL, viewport, timestamp, stats

   {
     "url": "https://www.costco.com/...",
     "viewport": { "width": 1920, "height": 1080 },
     "stats": {
       "totalNodes": 2847,
       "salienceCounts": { "high": 12, "med": 89, "low": 2746 }
     }
   }

2. Structural skeleton – high‑level layout blocks (the page’s “table of contents”)

   {
     "layoutBlocks": [
       { "role": "header",   "contains": ["logo", "nav", "search"] },
       { "role": "sidebar", "contains": ["filters", "categories"] },
       { "role": "main",    "contains": ["product-grid"] }
     ]
   }

3. Element‑specific data – selectors, text, visibility, interaction state

   {
     "btn042": {
       "selector": "button.add-to-cart",
       "text": "Add to Cart",
       "actions": ["clickable"],
       "styles": { "visible": true, "disabled": false }
     }
   }

4. Spatial relationships – semantic positioning (no pixel coordinates)

   {
     "btn042": {
       "inside": "card-product-123",
       "below": "price-display",
       "rightOf": "quantity-selector"
     }
   }

5. Salience tagging – guides the model’s focus

   • High: primary actions, main content, user inputs
   • Medium: secondary navigation, supporting info
   • Low: wrappers, containers, decorative elements

   Example map:

   PAGE STRUCTURE:
   ├── Header (logo, nav, search, cart)
   ├── Sidebar (filters)
   └── Main
       ├── Breadcrumbs
       ├── Product Grid (24 items)
       └── Pagination

---

Compression results (representative)

On complex pages, SiFR makes LLM workflows practical where raw HTML often exceeds context windows.

---

Getting started

SiFR is implemented in Element to LLM, a free browser extension:

• Chrome Web Store (also works on Arc, Brave, Edge)
• Firefox Add‑ons

Stress‑test the format

Try capturing snapshots of:

• https://www.costco.com – a realistic, framework‑heavy enterprise UI
• https://www.arngren.net – extreme visual density and chaotic layout

After capturing, compare compression ratios and see whether your LLM can reason about the structure. If you encounter a site where SiFR struggles, share the findings—this data helps improve the spec.

---

Use cases with structured runtime UI context

• Debug layouts – paste JSON to spot z‑index / visibility / layout issues
• Generate selectors – create Playwright/Cypress tests based on real DOM structure
• Autonomous navigation – agents understand “where to click” without screenshots
• Recreate components – translate UI structure into React/Tailwind scaffolds

---

Specification status

• Current version: v2 (strict, versioned, designed for automation pipelines)
• Open specification – contributions and feedback are welcome.

If you’re building LLM‑powered UI tools, let us know:

• What feels missing?
• What feels redundant?
• What would make this more useful in your workflow?

---

Related posts

• Taking a “fine” signup form and making it work
• a11y starts with runtime context
• QA That Speaks JSON

---

More links

• Element to LLM – Chrome Web Store
• Element to LLM – Firefox Add‑ons

Found a site that breaks SiFR? Drop it in the comments – that’s the fastest way to improve the spec.