Skip to content
By NiLu Updated

Web agent-ready — machine representations: HTML, Markdown, and chrome

How HTML, negotiated Markdown, and page chrome affect agents and the cost of serving bots — what's standard and what's still a hypothesis.

In brief

  • Agents and crawlers typically read the DOM, the accessibility tree, or Markdown — not the "decorated" page that a human sees.
  • Content negotiation (Accept: text/markdown) is a standard HTTP mechanism, implemented by several platforms — not one vendor's invention.
  • HTML-to-Markdown size reductions in published cases range from ~20% to ~99%, but these are examples, not web-wide averages.
  • Adoption of Markdown negotiation is still rare, and not every agent asks for it — measure your own traffic before investing.

What an agent actually reads

A browser agent rarely “sees” the page the way a human does. According to web.dev, AI tools operate on machine representations — screenshot, raw HTML/DOM, and the accessibility tree — often combined. The accessibility tree summarizes each element’s roles, names, and states and ignores the visual noise of CSS; that’s why the same article recommends semantic interactive HTML (button, a) instead of decorated div/span, correct label associations, and stable layouts. [1]

Google Search Central’s guide to generative AI features reinforces this point: semantic HTML is good practice for humans and screen readers; perfect HTML validity is not required; and browser agents can use screenshot, DOM, and the accessibility tree. The documentation itself calls agent preparation optional — something to do “if there’s extra time,” not a requirement. [2]

How to serve a machine representation

The underlying mechanism is plain HTTP: content negotiation. The client sends an Accept header indicating the preferred format; the server chooses the representation and signals caches with Vary, according to MDN documentation. [3] Around this mechanism, several platforms document similar patterns: Vercel publishes a guide to serving Markdown and HTML at the same URL via Accept: text/markdown (rewrites + route handler); Fastly describes an edge service that delivers HTML to browsers and Markdown to known AI crawlers or explicit requests, always varying by Accept and User-Agent; and Cloudflare documents an equivalent zone-level conversion, stripping header, navigation, footer, and scripts from the Markdown body while preserving JSON-LD at the end of the document. [4] [5] [6]

None of these implementations is a single industry standard — they are the same idea (content negotiation) applied by different vendors. Publishing static Markdown files (such as /index.md) or an llms.txt index is another strategy, used for example in Cloudflare’s own Docs for agents documentation — simpler to operate, but with the risk of the Markdown drifting from the canonical HTML over time. [7]

Published cases: how much this matters

Publishers report significant size reductions when swapping HTML for Markdown — but these are case studies, not a measured web-wide average. Cloudflare cited, in a blog post, an example with about 16.180 tokens in HTML versus 3.150 in Markdown (~80% less) [8]; Checkly measured, in its own documentation, about 615 KB / 180 thousand tokens in HTML versus 2,3 KB / 478 tokens in Markdown (~99,6% less) [9]; and Fastly qualitatively describes typical articles compressing to 20–30% of the original HTML size [5].

Markdown negotiation only helps if something asks for it: Checkly itself tested seven coding agents and found only three (Claude Code, Cursor, and OpenCode) sending an Accept header requesting text/markdown — the other four (Codex, Gemini CLI, GitHub Copilot, and Windsurf) did not prioritize Markdown in the observed sample. [9] A Cloudflare Radar snapshot, cited in the company’s own Agent Readiness blog post, measured that Accept: text/markdown negotiation passes on only 3,9% of the high-traffic sites scanned — a point-in-time snapshot of low adoption, not a stable baseline. [10]

Chrome, cost, and bot traffic

The page’s “chrome” — menu, footer, scripts, decoration — has no semantic value for an agent, but it consumes context tokens on the agent’s side and bandwidth bytes on the side of whoever serves the site. Edge networks report growing volumes of AI bot traffic: Fastly recorded AI crawlers accounting for nearly 80% of the AI bot traffic observed in its mid-2025 analysis (with Meta, Google, and OpenAI as the largest shares), and general automated bot traffic at 37% of the network’s own activity in the period; Akamai reported growth of approximately 300% year-over-year in AI bot activity on its platform. [11] [12]

Reducing chrome could, in theory, reduce bytes per bot request — but this is an operational hypothesis, not a proven financial return. Absent a public study of origin/egress cost measured before and after a representation change, the practical advice is to measure bytes, status, and user-agent in your own logs before promising savings.

llms.txt: index or showcase

llms.txt works as a sitemap-style index, intended for tools that already consume it — that’s how Cloudflare itself describes it. [10] But the official Google Search Central guide lists llms.txt and special AI markup among the misconceptions to avoid: it is not a requirement to appear in generative AI features in Search. [2] An Ahrefs study, analyzing about 137 thousand domains, found that 28% published a valid llms.txt, but 97% of those files received no requests at all in the measured month — and, of those that did receive traffic, 96% of the requests came from bots, not editorial reading. [13]

This doesn’t prove that llms.txt is useless forever; it proves that, today, publishing it rarely changes anything observable. Treat it as an optional convenience for tools that already consume the format — not as a priority deliverable.

Uncertainties

  • There is no public study that isolates the effect of accessibility (WCAG) conformance on agent task success, beyond the qualitative argument from Google/web.dev.
  • It is not known which agents and crawlers will reliably send Accept: text/markdown — the Checkly sample (3 of 7) is a point-in-time snapshot.
  • There is no published operational data on the real cost of maintaining a static Markdown tree in parallel with HTML, including the risk of drift.
  • There is no public case study measuring bandwidth or origin savings in concrete figures, before and after serving negotiated Markdown to bots.

References

  1. web.dev — Build agent-friendly websites. web.dev. Accessed August 07, 2026.
  2. Google Search Central — Optimizing your website for generative AI features on Google Search. Google. Accessed August 07, 2026.
  3. MDN — Content negotiation. MDN. Accessed August 07, 2026.
  4. Vercel — How to serve documentation for agents. Vercel. Accessed August 07, 2026.
  5. Fastly — Give AI Agents the Markdown They Actually Want. Fastly. Accessed August 07, 2026.
  6. Cloudflare — Markdown for Agents (docs). Cloudflare. Accessed August 07, 2026.
  7. Cloudflare — Docs for agents. Cloudflare. Accessed August 07, 2026.
  8. Cloudflare — Introducing Markdown for Agents. Cloudflare. Accessed August 07, 2026.
  9. Checkly — The Current State of Content Negotiation for AI Agents. Checkly. Accessed August 07, 2026.
  10. Cloudflare — Introducing the Agent Readiness score. Cloudflare. Accessed August 07, 2026.
  11. Fastly — New Fastly threat research reveals AI crawlers make up almost 80% of AI bot traffic. Fastly. Accessed August 07, 2026.
  12. Akamai — AI Bots Threaten Foundation of Web-Based Business Models. Akamai. Accessed August 07, 2026.
  13. Ahrefs — We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read. Ahrefs. Accessed August 07, 2026.