Before the fetch: how AI agents discover what they can do with your site

We have spent a few posts now tracing the single request that finally lands on your server. It is a plain fetch, no JavaScript, carrying a name like Claude-User, and it only runs if the page in front of it was legible enough to be worth reading.

That fetch is the last step. Before it happens, an agent (or the harness acting for it) usually asks two other questions, somewhere else entirely, against files and endpoints most website owners have never looked at.

Am I allowed to touch this site at all, and if so, for what? And separately: what's actually here, and how do I call it, rather than just scrape it?

Neither question is new in spirit. Search engines have asked the first one since 1994. But the answers are being rewritten right now, specifically for agents. And the second question barely existed before 2024 at all. Increasingly, a site or a service does not just want to be read. It wants to be called, the way you'd call an API, with a defined set of capabilities instead of a pile of HTML to scrape and hope for the best.

This post is a map of both layers: the permission conventions (robots.txt and what is being bolted onto it) and the capability conventions (llms.txt, .well-known/, and the newer discovery and invocation protocols: Agentic Resource Discovery (ARD), the Model Context Protocol (MCP), and Agent2Agent (A2A)). We'll close with a layer that looks similar but is not actually a protocol at all: the "GEO/AEO/SEO 3.0" conversation, built on search engine optimization (SEO), about getting recommended by the agents that pass all of this.

"Am I allowed here?" - robots.txt grows teeth#

The Robots Exclusion Protocol is almost as old as the commercial web. Martijn Koster proposed it in 1994, informally, after crawlers caused real server-load problems. For 28 years it ran as a de facto standard with no formal spec at all, every crawler implementing its own interpretation of the edge cases, until the Internet Engineering Task Force (IETF) formalized it in 2022 as RFC 9309, one of the IETF's Request for Comments (RFC) standards-track documents.¹

The mechanics are unchanged by any of this AI-era activity. A compliant client fetches /robots.txt before anything else, matches the most specific User-agent group that applies to it, and resolves Allow/Disallow conflicts by longest matching prefix. What has changed is how many parties now show up in that file. Dozens of companies register their own crawler token so a site can allow or block them independently of ordinary search indexing: GPTBot, ChatGPT-User, and OAI-SearchBot from OpenAI; ClaudeBot, Claude-User, and Claude-SearchBot from Anthropic; Google-Extended, kept deliberately separate from Googlebot so a site can allow Search indexing while refusing AI training; CCBot from Common Crawl, which feeds a large share of foundation-model training sets; PerplexityBot, Applebot-Extended, Amazonbot, and others. One file went from a single on/off switch to a multi-party policy document.

Real sites are already taking visibly different positions. Reddit's robots.txt is a blanket Disallow: /, pointing crawlers instead at its Public Content Policy.² The New York Times writes its refusal in prose rather than a machine directive, naming Article 4 of the EU's Copyright Directive directly in the file's header comment.³ Cloudflare's, by contrast, explicitly allows eight named AI crawlers and adds a new line beneath them:

User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Anthropic-AI
Allow: /
 
# Content Signals - declare AI content usage preferences
Content-Signal: ai-train=yes, search=yes, ai-input=yes

That Content-Signal line is the actual extension worth understanding, because Allow/Disallow was only ever built to answer "can you fetch this," not "what can you do with it afterward." Cloudflare published the Content Signals Policy in September 2025, CC0-licensed, defining three independent signals: search, ai-input (retrieval-augmented generation, or RAG, and grounding use), and ai-train (training and fine-tuning), each yes, no, or simply absent.⁴ A no is written to carry legal weight, framed explicitly as "an express reservation of rights under Article 4" of that same EU directive, not just a polite request. Cloudflare has already pushed search=yes, ai-train=no into the managed robots.txt it serves across the 3.8 million-plus domains that opted into blocking AI training.⁴

The standards-track version of this is already underway at the IETF, in the AI Preferences (AIPREF) working group, chaired by Mark Nottingham, the same person who authored the .well-known specification below.⁵ Its charter is to standardize the vocabulary behind signals like Cloudflare's, define how that vocabulary attaches via existing mechanisms including robots.txt and well-known URIs, and specify how to reconcile conflicting preferences. No RFC has shipped yet. Content Signals is best read as a live preview of where that work is heading, not a finished standard in itself.

If you run a website, this layer is the one to get right first, because it is the one every other mechanism in this post assumes already ran. Agent identity is the layer that tells you who is asking; robots.txt and Content Signals are what let you answer, per name, whether they're welcome at all.

The agentic web's address book: .well-known/#

The next layer is not an AI concept, and that is exactly why it works. .well-known/ is a generic namespace-reservation convention, standardized as RFC 8615 in 2019 (obsoleting the original 2010 RFC 5785), so that any new site-wide metadata protocol has one predictable place to register a manifest instead of squatting on a top-level path and risking a collision.⁶ The Internet Assigned Numbers Authority (IANA) keeps the canonical registry of suffixes so specs can reserve a name without stepping on each other.⁷

A client that already knows the registered suffix just issues a plain GET against https://{host}/.well-known/{name}. .well-known itself defines nothing about the response; format, auth, and semantics belong entirely to whatever spec claimed that suffix. What is notable is how many AI-agent-facing specs built since 2023 chose this address book over inventing their own:

GitHub's own security.txt is a clean, working instance of the oldest and simplest entry in that table, a plain contact process for a human researcher to follow.⁸

The ai-plugin.json row is worth a second look, because it is a cautionary tale and a precedent at the same time. OpenAI launched it in March 2023 as arguably the first mainstream well-known-based AI agent discovery convention, letting ChatGPT call a site's API as a "plugin." Plugins were deprecated in 2024 in favor of GPTs and Actions, and fetching Klarna's old ai-plugin.json manifest today (Klarna was OpenAI's flagship 2023 plugin partner) still resolves rather than 404ing, but behind a WAF challenge rather than a clean response.⁹ The specific plugin format died. The pattern it popularized, a JSON manifest at a well-known path describing how to call a site, is exactly what A2A's agent-card.json and ARD's ard.json still use today.

How unsettled some of these entries still are is worth naming plainly. Checking agenticresourcediscovery.org, huggingface.co, and github.com for a live /.well-known/ard.json or its predecessor /.well-known/ai-catalog.json returned a 404 on all three during this research, including on domains run by ARD's own named contributors. The well-known paths are mature and uncontested; some of the content that is supposed to live at them is still, in places, not actually deployed yet.

"What's here, and how do I call it?"#

Knowing you're allowed in, and knowing where a manifest would live if one existed, still leaves the harder question open: once an agent is in, what is there to actually use?

llms.txt: a map, with no confirmed reader yet#

Jeremy Howard of Answer.AI proposed /llms.txt in September 2024 as a condensed, Markdown-native table of contents for a site, aimed squarely at the problem that a cluttered, nav-heavy HTML homepage is an inefficient way to hand a large language model (LLM) your most important content.¹⁰ The format is simple by design: an H1 with the site name, a required one-line blockquote summary, then H2-delimited sections of - [link](url): note bullets, with an ## Optional section marking what a context-constrained reader can skip.¹¹ A sibling file, llms-full.txt, inlines every linked page's full content into one document instead of linking out, meant for a build script or indexer rather than incremental fetching.

Cloudflare's [llms] is a fully spec-compliant example, complete with the required blockquote, and it pairs with something that makes the second hop genuinely cheap: every page is also available as clean Markdown by appending .md to its URL.¹² The Model Context Protocol's own docs site publishes one too, though it actually skips the required blockquote, a small, real compliance gap that shows this is a loose convention in practice, not something validated against the spec by anyone.¹³

MCP and A2A: the layer that actually calls something#

llms.txt and the well-known manifests above describe things. Model Context Protocol and Agent2Agent are what actually call them.

Anthropic proposed MCP in November 2024 to answer a narrower question than discovery: given a specific tool or data source, how does a client invoke it, over JSON-RPC 2.0, with a defined handshake for listing and calling tools.¹⁵ It has since been adopted by OpenAI and Google DeepMind as well, which is a large part of why ARD and A2A both treat it as a thing to point at rather than compete with. Google announced A2A in April 2025 and donated it to the Linux Foundation later that year; it solves a different problem, agent-to-agent interoperability, letting opaque agents from different vendors negotiate a task without exposing their internal state, with a server's identity and skills published at /.well-known/agent-card.json.¹⁶

If this tool-versus-model split sounds familiar, it's the same distinction we drew out when comparing how different harnesses implement web search and fetch: a model never touches a socket or calls an MCP server directly. It emits a structured request naming a tool and some arguments, and the harness is what actually opens the connection, whether that connection is a plain HTTP fetch or a JSON-RPC call into an MCP server.

ARD: finding the right resource before you call it#

Once there are thousands of MCP servers and A2A agents scattered across the web, a new problem shows up one level above either protocol: given a task, which resource, anywhere, can help, without stuffing every tool description that exists into a single context window?

Agentic Resource Discovery (ARD) is a pre-standardization proposal, currently at v0.91 (August 2026), scoped entirely to discovery, with no role in invocation itself.¹⁷ It does not replace MCP, A2A, or Skills; it is the search engine that points a client at one. A resource publishes an "ARD entry," a JSON-LD (JSON for Linked Data) object naming its type (an MCP server, an A2A agent, and so on) and, ideally, a handful of representative natural-language queries it can serve, via /.well-known/ard.json, an Agentmap: line in robots.txt, or a few other optional channels. A registry crawls those entries and exposes a search API; a client queries it, gets back a ranked url, and then invokes whatever that url points to through that resource's own native protocol, MCP or A2A, not through ARD itself.

The authorship is what makes this worth tracking rather than filing away as another early-stage spec: lead authors from Google, Microsoft, and Hugging Face, with contributors credited from AWS, Cisco, Databricks, GitHub, GoDaddy, Nvidia, Salesforce, and Snowflake.¹⁷ That is a wider coalition than most pre-standardization web specs gather before a first stable release, even though, as the dead .well-known/ard.json lookups above show, almost nothing is actually deployed against it yet.

How the chain actually runs, end to end#

None of this is a menu where a client picks one. It is a sequence, and ARD's own worked example makes the composition concrete: a user asks an agent's harness (ARD's own spec calls this role an "orchestrator") to book a flight and file the expense report. The harness queries its enterprise Agent Registry, which hands back an internal expense agent directly and a referral to a public registry for the flight-booking part. The harness follows that referral, finds a flight agent, then invokes the expense agent through MCP and the flight agent through A2A.¹⁷ ARD never touches execution. It only ever answered what exists and where.

What happens after either of those handoffs is its own small protocol in itself. An MCP client that hits an unauthenticated-required server gets a 401 pointing at /.well-known/oauth-protected-resource (RFC 9728), which names an authorization server, which in turn publishes its own metadata at /.well-known/oauth-authorization-server (RFC 8414); only after an OAuth 2.1 code exchange does actual tools/call traffic start, bearer token attached to every request.¹⁵ An A2A client, by contrast, reads a server's endpoint straight off its Agent Card and opens a session with a message/send call, getting back a stateful Task that moves through submitted, working, and a terminal state as the server works.¹⁶

That MCP auth handshake is worth connecting back to the provable-identity problem we wrote about separately: where Web Bot Auth lets a site cryptographically verify which agent is knocking at the plain-HTTP layer, MCP's OAuth flow is the same instinct, who is this and are they allowed, rebuilt for the tool-calling layer instead of the fetch layer. Different protocol, same underlying question, asked at a different point in the chain. Robots.txt and Content Signals sit entirely outside both of these chains, as a yes/no gate a crawler checks before any of this starts; unlike the discovery files, a robots.txt line never hands off into a live session at all.

The layer above all of this: GEO, AEO, and "SEO 3.0"#

Everything above has a GET request or a wire protocol behind it. This last layer doesn't, and it's worth naming precisely so it doesn't get confused with the rest of this post.

Generative Engine Optimization (GEO) is a real, independently-notable term with an academic origin: a 2023 paper by Pranjal Aggarwal and colleagues, presented at the 2024 ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), introducing it as a paradigm for improving content visibility in generative engine responses, with its own benchmark.¹⁸ Answer Engine Optimization (AEO) predates it, originally describing optimization for voice assistants and featured snippets, and is now used interchangeably with GEO by some vendors and distinctly by others.¹⁹ Wikipedia's own summary is blunt about the state of the terminology: "usage varies across practitioners, vendors, and publications, with no consensus definition."¹⁹ Other labels circulating for roughly the same cluster of practices include AIO (AI optimization), LLMO (LLM optimization), and plain "AI SEO."

"SEO 3.0" specifically, as a trademarked three-era framework, traces to one consulting brand and does not appear to have independent usage beyond its own source.²⁰ That doesn't make the underlying practices fake, it just means the label is a single consultant's entry in an active naming contest, not a settled category sitting alongside robots.txt or ARD in the table above.

The companies that actually run AI search have pushed back on the idea that any of this is a new discipline. Google's own guidance states its position plainly: "optimizing for generative AI search is optimizing for the search experience, and thus still SEO."²¹ Forrester analyst Nikhil Lai went further, writing that these terms are "significantly, but not fundamentally, different from SEO," and that their advocates "tend to exaggerate SEO and AEO's differences to carve a startup-sized hole in marketers' tech stacks."¹⁹

What is genuinely real underneath the naming fight is that the practice decomposes into the exact protocols already in this post: permissive AI-crawler rules and Content-Signal lines in robots.txt, a well-formed llms.txt, structured JSON-LD markup of the kind ARD's entries are built from, and plain legible, server-rendered content. GEO/AEO/"SEO 3.0" names a goal, getting cited or recommended by an agent, and a measurement layer on top (Search Console's AI performance reports, Bing's equivalent, third-party trackers). It is strategy wrapped around the protocols in this post, not another protocol to implement alongside them.

What this means if you run a website#

Stack all of this up and a website owner's actual to-do list is shorter than the research behind it:

• Decide your permissions on purpose, not by default. A blank robots.txt is a decision too, it just means every AI crawler that identifies itself honestly gets a yes by omission. Naming the crawlers you want and adding Content-Signal lines for training versus RAG use costs one file edit.
• Publish a real llms.txt if your docs are worth reading. Adoption on the reading side is still thin, but it costs little to produce and some real tools already consume it, coding agents especially.
• Don't expect .well-known/ard.json or an Agent Card to matter yet unless you're already building an MCP server or an A2A agent. Those are invocation-layer concerns for sites that expose a callable service, not a reading-layer concern for a typical content site.
• Treat GEO/AEO pitches as a reminder list, not a new checklist. If a vendor's "AI visibility audit" boils down to permissive crawler rules, an llms.txt, structured markup, and legible server-rendered HTML, that is this whole post compressed into a sales deck. Nothing wrong with the substance; just notice there's no new protocol hiding behind the name.

At Aandeg we're building Agent Assay, which requests your website the way an agent's harness does and reports whether what comes back is something an agent could actually read or call. The protocols in this post decide whether an agent is allowed to show up and what it finds when it asks around. What we check is what happens the moment it actually arrives.