Introducing Markdown for Agents

Source: Cloudflare Blog

The Evolving Landscape of Online Discovery

The way content and businesses are discovered online is changing rapidly.

• Past: Traffic primarily originated from traditional search engines, and SEO determined who got found first.
• Present: Traffic is increasingly coming from AI crawlers and agents that require structured data within the often‑unstructured web that was originally built for humans.

Why This Matters for Your Business

To stay ahead, you must go beyond focusing solely on human visitors and traditional SEO wisdom. Treat AI agents as first‑class citizens by:

1. Providing clear, machine‑readable structured data (e.g., JSON‑LD, schema.org).
2. Optimizing content for both humans and AI agents.
3. Monitoring AI‑driven traffic patterns and adjusting strategies accordingly.

Now is the time to adapt your digital strategy to the new reality where agents—not just people—determine visibility and success online.

Why markdown is important

Feeding raw HTML to an AI is like paying by the word to read packaging instead of the letter inside. A simple

## About Us

on a page in markdown costs roughly 3 tokens; its HTML equivalent

## About Us

burns 12–15 tokens, and that’s before you account for the “ wrappers, nav bars, and script tags that pad every real web page and have zero semantic value.

This blog post you’re reading takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown. That’s an 80 % reduction in token usage.

Markdown has quickly become the lingua franca for agents and AI systems as a whole. The format’s explicit structure makes it ideal for AI processing, ultimately resulting in better results while minimizing token waste.

The problem is that the Web is made of HTML, not markdown, and page weight has been steadily increasing over the years, making pages hard to parse. For agents, the goal is to filter out all non‑essential elements and scan the relevant content.

The conversion of HTML to markdown is now a common step for any AI pipeline. Still, this process is far from ideal: it wastes computation, adds costs and processing complexity, and, above all, it may not be how the content creator intended their content to be used in the first place.

What if AI agents could bypass the complexities of intent analysis and document conversion, and instead receive structured markdown directly from the source?

Convert HTML to markdown, automatically

Cloudflare’s network now supports real‑time content conversion at the source for enabled zones using content negotiation headers. When AI systems request pages from any website that uses Cloudflare and has Markdown for Agents enabled, they can express a preference for text/markdown in the request. Cloudflare will automatically and efficiently convert the HTML to markdown on the fly, when possible.

How it works

To fetch the markdown version of any page from a zone with Markdown for Agents enabled, the client must add the Accept negotiation header with text/markdown as one of the options. Cloudflare detects this, fetches the original HTML from the origin, converts it to markdown, and serves the result to the client.

cURL example

curl https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ \
  -H "Accept: text/markdown"

Workers (TypeScript) example

const r = await fetch(
  "https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/",
  {
    headers: {
      Accept: "text/markdown, text/html",
    },
  },
);
const tokenCount = r.headers.get("x-markdown-tokens");
const markdown = await r.text();

Popular coding agents such as Claude Code and OpenCode already send these Accept headers with their requests. The response is now formatted in markdown—nothing more to do.

Sample HTTP response

HTTP/2 200
date: Wed, 11 Feb 2026 11:44:48 GMT
content-type: text/markdown; charset=utf-8
content-length: 2899
vary: accept
x-markdown-tokens: 725
content-signal: ai-train=yes, search=yes, ai-input=yes

---
title: Markdown for Agents · Cloudflare Agents docs
---

What is Markdown for Agents

The ability to parse and convert HTML to Markdown has become foundational for AI.

Note: We include an x-markdown-tokens header with the converted response that indicates the estimated number of tokens in the Markdown document. You can use this value in your flow—for example, to calculate the size of a context window or to decide on your chunking strategy.

How It Works

Content Signals Policy

During our last Birthday Week, Cloudflare announced Content Signals—a framework that allows anyone to express their preferences for how their content can be used after it has been accessed.

When you return Markdown, you want to make sure your content is being used by the Agent or AI crawler. That’s why Markdown‑for‑Agents responses include the header

Content‑Signal: ai‑train=yes, search=yes, ai‑input=yes

which signals that the content can be used for AI training, search results, and AI input (including agentic use). Markdown for Agents will provide options to define custom Content Signal policies in the future.

Check our dedicated Content Signals page for more information.

Try It with the Cloudflare Blog & Developer Documentation

We enabled this feature in our Developer Documentation and our Blog, inviting all AI crawlers and agents to consume our content using Markdown instead of HTML.

Request the blog with Markdown:

curl https://blog.cloudflare.com/markdown-for-agents/ \
  -H "Accept: text/markdown"

Sample response:

---
description: The way content is discovered online is shifting, from traditional search engines to AI agents that need structured data from a Web built for humans. It’s time to consider not just human visitors, but start to treat agents as first‑class citizens. Markdown for Agents automatically converts any HTML page requested from our network to markdown.
title: Introducing Markdown for Agents
image: https://blog.cloudflare.com/images/markdown-for-agents.png
---

# Introducing Markdown for Agents

The way content and businesses are discovered online is changing rapidly. In the past, traffic originated from traditional search engines and SEO determined who got found first. Now the traffic is increasingly coming from AI crawlers and agents that demand structured data within the often‑unstructured Web that was built for humans.

...

Other Ways to Convert to Markdown

If you’re building AI systems that require arbitrary document conversion from outside Cloudflare, or Markdown for Agents is not available from the content source, we provide other conversion options:

• Workers AI – AI.toMarkdown() supports multiple document types (not just HTML) and summarization.
• Browser Rendering – /markdown REST API converts a page rendered in a real browser before returning Markdown.

End of document

Tracking markdown usage

Anticipating a shift in how AI systems browse the Web, Cloudflare Radar now includes content‑type insights for AI bot and crawler traffic—both globally on the AI Insights page and on the individual bot information pages.

The new content_type dimension and filter shows the distribution of content types returned to AI agents and crawlers, grouped by MIME type category.

You can also view the requests for Markdown filtered by a specific agent or crawler. Below are the requests that return Markdown to OAI‑Searchbot, the crawler used by OpenAI to power ChatGPT’s search:

This new data will allow us to track the evolution of how AI bots, crawlers, and agents are consuming Web content over time. As always, everything on Radar is freely accessible via the public APIs and the Data Explorer.

Start using today

To enable Markdown for Agents for your zone, log into the Cloudflare dashboard, select your account, choose the zone, look for Quick Actions, and toggle the Markdown for Agents button to enable. This feature is available today in Beta at no cost for Pro, Business, and Enterprise plans, as well as SSL for SaaS customers.

You can find more information about Markdown for Agents in our Developer Docs. We welcome your feedback as we continue to refine and enhance this feature. We’re curious to see how AI crawlers and agents navigate and adapt to the unstructured nature of the Web as it evolves.