Web agent-ready — what changes when part of the audience isn't human
Scope: Three-layer operational map — discover, cite and execute; what changes for those who operate a website when bots and agents are part of the audience; decision priority stack.
Executive summary
Three layers — discover, cite and execute — better describe 2026 than reducing everything to 'SEO with more schema'. It is an operating model tested against available evidence, not a closed scientific thesis.
Complete executive summary
- Part of a website's traffic and audience is non-human — crawlers, bots and, still experimentally, agents acting on behalf of a user. Vendor telemetry (Imperva) is neither a census of the web nor evidence of demand for AI agents.
- Three distinct jobs — discover, cite and execute — organize priority and budget better than treating everything as 'SEO with more schema'. For Google, optimizing for AI capabilities in Search is still SEO: no special markup or requirement for llms.txt.
- Citation is not a visit: click-through rates fall when an AI summary appears (Pew), and publishers report pressure on referrals and zero-click search (Digiday). Agent execution (WebMCP) is an origin trial and community draft — monitor it, but do not commit budget.
System map and relations
Audience: Website managers and owners, builders, technical editors, and operators.
Out of scope
- A detailed comparison of formats, Markdown negotiation, and HTML chrome remains in the specialized Representations Dossier.
- A more detailed discussion of SEO vs GEO, CTR, and zero-click remains in the specialized SEO vs GEO Dossier; execution via WebMCP remains scoped, not implemented.
website audience (people + crawlers + bots +, experimentally, agents)
↓
Discover — crawling, indexing, snippet eligibility / AI features (same basics as SEO)
↓
Cite — retrieval and synthesis with attribution in a generative response (not a ranking position)
↓
Execute — agent calls actions on website via WebMCP (origin trial / community draft)
↓
cross-cutting layer: representation (semantic HTML/a11y, negotiated Markdown) supports Discover and Cite
↓
priority stack: SEO foundation → clean representations → crawl policy → actual KPIs → execution on watchSpecialized Dossiers
DATA
Claims with explicit sources, measurement context, limitations, and confidence.
DATA · fact
WA-OV-D01 · Google: Optimizing for AI in Search is still SEO
The Google Search Central guide to generative AI capabilities states that optimizing for these capabilities is optimizing for Search, under the same core quality and ranking systems; it identifies reliance on llms.txt, special AI markup, splitting content into chunks, or rewriting text solely for agents as misconceptions; structured data is not required for generative AI capabilities.
Limitations
DATA · fact
WA-OV-D02 · Eligibility for AI features requires no extra technical requirement
Google Search Central's 'AI features and your website' documentation states that there is no additional technical requirement beyond being indexed and eligible for snippet, and recommends keeping important content in textual form.
Limitations
DATA · fact
WA-OV-D03 · People-first and usual technical requirements extend to AI Overviews
Search Central Blog's 'Succeeding in AI search' article reiterates that people-first advice, page experience and Search standard technical requirements (crawling, indexing, snippet eligibility) extend to AI Overviews and AI Mode; Structured data is only useful when it matches visible content.
Limitations
DATA · fact
WA-OV-D04 · Chrome describes two stages of the 'agentic web'
The Chrome for Developers blog 'A developer toolkit to make your website agent-ready' separates two stages: agents that search (searching, where the fundamentals of SEO continue to apply) and agents that use the website directly (using, focusing on accessibility tree, layout stability and WebMCP); the Lighthouse Agentic Browsing score is treated as informational while patterns still emerge.
Limitations
- Browser implementer language, not Search standard nor scientific proof of the layering model.
DATA · fact
WA-OV-D05 · W3C TAG discusses distinguishing humans, bots and agents in Web architecture
The W3C Technical Architecture Group recorded, at the 2026 March meeting in London, a discussion about how Web architecture should distinguish humans, bots and agents acting on behalf of users — an ongoing debate, with no final Finding published in the post.
Limitations
- Ongoing institutional discussion; it is not a closed conclusion of W3C.
DATA · estimate
WA-OV-D06 · Imperva records higher-than-human automated traffic, with caveats
Imperva's Bad Bot 2025 report, based on the vendor's own network at 2024, recorded automated traffic (51%) exceeding human traffic (49%), with 37% classified as a malicious bot and 14% as a 'good' bot.
Limitations
- It is not an independent census of the web; it is a sample from a security vendor's network.
- 'Automated' is not synonymous with 'AI agent'; it does not distinguish agents acting on behalf of users from ordinary crawlers and scrapers.
DATA · fact
WA-OV-D07 · Content negotiation is standard HTTP mechanism
MDN describes content negotiation in HTTP as the mechanism for serving different representations of the same resource at the same URI: the client sends Accept headers and the server responds accordingly, flagging caches with Vary.
Limitations
DATA · fact · estimate
WA-OV-D08 · Vercel and Fastly document serving Markdown to agents via negotiation
Vercel documents serving Markdown and HTML at the same URL via Accept: text/markdown (rewrites + route handler); Fastly describes serving HTML to browsers and Markdown to crawlers from AI or Accept requests: text/markdown, with Vary, estimating a typical article in Markdown with about 20–30% the size of the equivalent HTML.
Limitations
- The Fastly size estimate is an illustrative case, not a web average.
DATA · estimate · comparison
WA-OV-D09 · Adoption of negotiated Markdown remains rare
Checkly measured that only 3 of 7 tested coding agents sent Accept asking for text/markdown; a snapshot Radar cited on Cloudflare's Agent Readiness blog found Markdown negotiation (Accept: text/markdown) working on only 3,9% from a sample of high-traffic sites scanned.
Limitations
- Small and specific samples; they do not measure the entire web or all existing agents.
DATA · estimate
WA-OV-D10 · 97% of published llms.txt files are not read
A study by Ahrefs, with a sample of around 137 thousand websites, found that 97% of the published llms.txt files did not receive any requests in the month analyzed.
Limitations
- It does not measure the intent of every existing AI agent or future periods.
DATA · estimate
WA-OV-D11 · Paper that defined GEO reports gain 'of up to 40%' in laboratory environment
The paper that defined the term GEO (Generative Engine Optimization), by Aggarwal et al., reported a relative gain in visibility 'up to 40%' in the authors' own experimental environment, with specific pipeline (Google's top-5 reprocessed by a GPT-3.5-style model) — not in a public ranking product.
Limitations
- It is not a public ranking API or business return prediction.
DATA · fact
WA-OV-D12 · ChatGPT search retrieves and synthesizes with attribution, without public scoreboard
OpenAI describes ChatGPT search as a tool that searches the web and returns answers with links to sources, with a Sources sidebar; the documentation does not publish a public citation position scoreboard.
Limitations
DATA · estimate
WA-OV-D13 · CTR drops when an AI summary appears
Pew Research Center found, in a correlational analysis, organic link click rates of about 8% when an AI summary appeared in Google results, versus about 15% when it did not; only about 1% of users clicked on links within the summary itself.
Limitations
- Correlation; does not establish universal causality.
DATA · fact
WA-OV-D14 · OpenAI distinguishes training, search, and real-time-use bots
The crawlers documentation for OpenAI distinguishes GPTBot (training), OAI-SearchBot (indexing for search), and ChatGPT-User (real-time user search), with separate robots.txt controls for each role.
Limitations
DATA · estimate
WA-OV-D15 · AI crawlers dominate the AI bot traffic observed by an edge vendor
Fastly (Threat Insights research) stated that AI crawlers accounted for nearly 80% of the AI bot traffic observed in its period analysis.
Limitations
- Network sample from a vendor, not a web census.
DATA · fact
WA-OV-D16 · WebMCP is origin trial in Chrome and Draft Community Group Report in W3C
The Chrome for Developers documentation describes WebMCP as proposed in origin trial from Chrome 149, with relevant limitations — requires an open browser context, does not work without a UI (headless), and discovering capabilities requires visiting the website directly; The W3C Web Machine Learning Community Group draft explicitly states that it is a Draft Community Group Report, not a W3C Standard nor part of the Standards Track.
Limitations
- Experimental proposal; It is not a stable cross-browser standard.
DATA · estimate · comparison
WA-OV-D17 · Publishers report pressure from referral and zero-click associated with AI
Reports from Digiday, citing data from Digital Content Next (DCN), reported approximately -10% year-over-year median referral traffic from Google between publishers (wide range from -1% to -25%) associated with AI Overviews; another report from Digiday, citing Similarweb, found zero-click in news searches rising from around 56% to 69%, with traffic from AI platforms growing without offsetting this loss.
Limitations
- Specialized press reporting on specific cohorts; does not establish universal causality nor generalize to every publisher.
DATA · estimate
WA-OV-D18 · Citation recall in generative search engines is fragile
A 2023 study (Liu et al.) measured recall citation per sentence around 51,5% in the generative search engines evaluated — nearly half of the sentences generated without an adequate verifiable citation in the snapshot studied.
Limitations
- Snapshot of a specific moment; evaluated mechanisms may have changed since then.
INFO
INFO · derived synthesis
WA-OV-I01 · Three layers organize jobs that 'SEO with schema' collapses together
Discover, cite and execute separate distinct jobs, evidence and KPIs that are hidden when everything is treated as 'SEO with more schema'; Chrome's stage language (searching/using) and W3C TAG's architecture discussion (humans/bots/agents) support the industry vocabulary but do not prove the model as science — it remains an editorial hypothesis tested against evidence.
derivedFrom: WA-OV-D01, WA-OV-D02, WA-OV-D04, WA-OV-D05, WA-OV-D11, WA-OV-D12, WA-OV-D16
INFO · derived synthesis
WA-OV-I02 · Discover is eligibility under established fundamentals, not special markup
The Discover layer is covered by the same core quality systems and ranking as Search: snippet indexing and eligibility, with people-first content — no special AI markup, no requirement for llms.txt, and no requirement for perfect HTML.
INFO · derived synthesis
WA-OV-I03 · Cite has a laboratory method and documented UX, not public scoreboard
The Cite layer combines a laboratory method (GEO) with a documented attribution UX (Sources), but there is no public citation position leaderboard; citation does not equal visit — click-through rates drop with AI summaries, publishers report pressure from referral/zero-click, and the quality of the citation itself is fragile.
derivedFrom: WA-OV-D11, WA-OV-D12, WA-OV-D13, WA-OV-D17, WA-OV-D18
INFO · derived synthesis
WA-OV-I04 · Clean representation is cross-cutting, but adoption of negotiated Markdown is still rare
Serving a clean representation — content negotiation HTTP, HTML semantic, Markdown when the stack supports it — is useful for discovery and citing, but still the exception: few agents ask for Markdown today, and most high-traffic sites scanned don't respond to Accept: text/markdown.
INFO · derived synthesis
WA-OV-I05 · Execute (WebMCP) remains experimental and outside critical roadmap
WebMCP is origin trial in Chrome and Draft Community Group Report in W3C, with relevant operational limitations; the Execute layer should be under observation, not guide budget today.
derivedFrom: WA-OV-D16
INFO · derived synthesis
WA-OV-I06 · Decision Priority Stack
The evidence supports an order of priority: (1) foundation of SEO and content people-first; (2) clean representations; (3) policy of crawl separating training bots from search/citation bots; (4) Search Console real metrics, traffic and bot bytes, not vanity llms.txt; (5) execution by agents as observation, not a roadmap bet.
derivedFrom: WA-OV-D01, WA-OV-D02, WA-OV-D03, WA-OV-D07, WA-OV-D10, WA-OV-D13, WA-OV-D14, WA-OV-D15, WA-OV-D16
INFO · derived synthesis
WA-OV-I07 · llms.txt is weak 'AI presence' KPI
Google does not require llms.txt for AI features in Search, and most published files are never read — publishing a llms.txt 'because everyone has it' is not a reliable indicator of priority or outcome.
INFO · derived synthesis
WA-OV-I08 · Automated traffic pressures crawl policy and cost, does not prove demand for execution
Telemetry from vendor on automated traffic (with caveats) and on AI crawlers' share in bot traffic drives crawl policy and bandwidth/origin cost sizing — not evidence that agents are ready to perform actions on the site nor does it justify prioritizing WebMCP.
Uncertainties
UNCERTAINTY · não mensurado publicamente
WA-OV-UNC-01 · Whether the three-tier model improves budget decisions
What we know: The three-tier model organizes jobs and evidence, but there is no usage study or editorial testing that confirms whether it improves budget decisions in practice, or just organizes the narrative without changing results.
Limitations
Editorial treatment: Treat the model as a working hypothesis tested against available evidence, not as a closed or scientifically proven framework.
Confidence: high na lacuna
Last reviewed: 2026-08-07
UNCERTAINTY · não documentado publicamente
WA-OV-UNC-02 · Support Link Selection Criteria in AI Responses
What we know: Google documents eligibility (indexed + eligible for snippet) for the Discover tier, but it doesn't publish the list of exact criteria it uses to choose which links appear in support of generative AI features.
Limitations
Editorial treatment: Do not invent a list of citation factors; treat link selection as a documented gap.
Last reviewed: 2026-08-07
UNCERTAINTY · sem previsão pública
WA-OV-UNC-03 · When WebMCP is no longer experimental
What we know: WebMCP is in origin trial in Chrome and as Draft Community Group Report in W3C; There is no public timeline for becoming Standards Track or stable cross-browser standard.
Limitations
Editorial treatment: Keep Execute as a layer under observation until the status changes are documented by the responsible bodies themselves.
Last reviewed: 2026-08-07
UNCERTAINTY · não mensurado publicamente
WA-OV-UNC-04 · Converting synthetic 'AI visibility' into real result
What we know: There is no public study that converts synthetic 'AI viewability' metrics (such as scores from third-party tools) into actual impressions, clicks or revenue for publishers.
Limitations
Editorial treatment: Prefer first-hand metrics (Search Console, traffic, conversion) to 'citation score' dashboards without method transparency.
Last reviewed: 2026-08-07
UNCERTAINTY · não desagregado publicamente
WA-OV-UNC-05 · Actual composition of 'automated' traffic
What we know: Telemetry such as Imperva does not separate, within the traffic classified as automated, which fraction is an agent acting on behalf of a user and which is a crawler or common scraper.
Limitations
Editorial treatment: Do not use automated traffic participation as proof of agent-readiness or to budget for 'agent presence'.
Last reviewed: 2026-08-07
Methodology
- DATA
- DATA records verifiable claims with explicit source, measurement context, limitations, and confidence.
- INFO
- INFO records synthesis or interpretation and always declares the IDs of the claims from which it derives.
- Uncertainty
- Uncertainty registers a gap as an editorial result; it remains explicit and is not filled with a fragile estimate.
- Update
- A revision can correct or update the projection only from the canonical Dossier with editorial authorization; Material changes increment the public version and are recorded in the corrections history.
- Public version
- 1.0.0
- Last verified
- 2026-08-07
- Review after
- 2026-09-06
Public corrections
- No public corrections recorded.
References
-
src-wa-google-ai-optimization-guide· Google Search Central — Optimizing your website for generative AI features on Google Search. Google. Accessed . -
src-wa-google-ai-features-website· Google Search Central — AI features and your website. Google. Accessed . -
src-wa-google-succeeding-ai-search· Google Search Central Blog — Succeeding in AI search. Google. Accessed . -
src-wa-chrome-agent-ready-toolkit· Chrome for Developers — A developer toolkit to make your website agent-ready. Chrome for Developers. Accessed . -
src-wa-w3c-tag-london-2026· W3C — The W3C TAG Meeting in London, March 2026. W3C. Accessed . -
src-wa-imperva-bad-bot-2025· Imperva — 2025 Bad Bot Report. Imperva. Accessed . -
src-wa-mdn-content-negotiation· MDN — Content negotiation. MDN. Accessed . -
src-wa-vercel-docs-for-agents· Vercel — How to serve documentation for agents. Vercel. Accessed . -
src-wa-fastly-markdown-agents· Fastly — Give AI Agents the Markdown They Actually Want. Fastly. Accessed . -
src-wa-checkly-content-negotiation-2026· Checkly — The Current State of Content Negotiation for AI Agents. Checkly. Accessed . -
src-wa-cloudflare-agent-readiness· Cloudflare — Introducing the Agent Readiness score. Cloudflare. Accessed . -
src-wa-ahrefs-llmstxt-study· Ahrefs — We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read. Ahrefs. Accessed . -
src-wa-arxiv-geo-2311-09735· Aggarwal et al. — GEO: Generative Engine Optimization. arXiv. Accessed . -
src-wa-openai-chatgpt-search· OpenAI — Introducing ChatGPT search. OpenAI. Accessed . -
src-wa-pew-ai-summary-ctr-2025· Pew Research Center — Google users are less likely to click on links when an AI summary appears in the results. Pew Research Center. Accessed . -
src-wa-openai-crawlers-docs· OpenAI — Crawler / bot documentation. OpenAI. Accessed . -
src-wa-fastly-ai-crawlers-press· Fastly — New Fastly threat research reveals AI crawlers make up almost 80% of AI bot traffic. Fastly. Accessed . -
src-wa-chrome-webmcp-docs· Chrome for Developers — WebMCP. Chrome for Developers. Accessed . -
src-wa-w3c-webmcp-cg-draft· W3C Web Machine Learning Community Group — WebMCP API (Draft Community Group Report). W3C WebML CG. Accessed . -
src-wa-digiday-dcn-referral-drop· Digiday — Google AI Overviews linked to publisher referral traffic drops, new data shows. Digiday. Accessed . -
src-wa-digiday-similarweb-zero-click· Digiday — In graphic detail: AI platforms are driving more traffic, but not enough to offset zero-click search. Digiday. Accessed . -
src-wa-arxiv-verifiability-2304-09848· Liu et al. — Evaluating Verifiability in Generative Search Engines. arXiv. Accessed .