I wanted AI bots reading my site. My robots.txt was telling them to leave — and it wasn't even mine.

By Daniel S. · July 23, 2026

TL;DR: Cloudflare shipped a scanner — isitagentready.com — that grades how readable your site is for AI agents. This site scored 29/100, and the embarrassing part wasn't the low number. It was the category I passed: full marks for "AI bot rules in robots.txt" — rules that were blocking every major AI crawler, on a site whose entire subject is AI, published by a robots.txt I never wrote. A dashboard toggle had been speaking for me. Here's the untangling, what actually moved the score to 43/100 on a free-plan static site, and which points a static blog honestly cannot buy.

The green check that was working against me

Run the scanner against your domain and you get five categories. Mine came back: Discoverability 2/4, Content 0/1, Bot Access Control 2/2, API/MCP discovery 0/7.

That 2/2 looked like the one thing I had right. Then I read why it passed: "Found rules for AI bots: gptbot, google-extended, ccbot, bytespider, applebot-extended, amazonbot…"

I never wrote rules for any of those bots. My robots.txt in the repo is four lines: allow everything, here's the sitemap. What the internet actually saw was that file with a Cloudflare-managed block prepended — Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended and friends, plus a Content-Signal: ai-train=no — courtesy of a "Managed robots.txt" toggle in the dashboard's AI-crawl settings.

The scanner's check only asks whether rules exist, not which direction they point. So I was collecting points for a policy that was the exact opposite of my intent. A green check has exactly the value of its discriminating power: this one couldn't tell "welcoming AI bots" from "blocking them all," so it had none.

This is the same lesson as config vs enforcement, with a twist that took me a minute to appreciate: the config wasn't mine. If a managed layer can speak for your site, then reading your own repo tells you nothing about what your site is saying. Assert the served state — curl yourdomain.com/robots.txt — not the file you committed.

The toggle maze, mapped so you don't have to

Untangling this took a tour through settings that are mid-migration and partly deprecated. The map, as of this week:

• Bot Fight Mode — on free plans, injects a JS-detection script into every page. On a static site with a strict CSP the script is blocked anyway, so it was pure console noise. Off.
• Managed robots.txt — the layer that was speaking for me. Off. The served robots.txt reverted to the repo's file within a minute.
• "Block AI bots" — the enforcement sibling (network-level, robots.txt-independent), already carrying a deprecation date. Mine read Allow. If yours reads Block, fixing robots.txt changes nothing — polite signs don't matter when the door is locked.
• "Configure block response" — only shapes what blocked crawlers see. Irrelevant in an allow posture; leave it.

One honest residual: the better part of an hour after everything read "off," the wire was still showing the injected bot-detection script on cache-busted requests marked as uncached. Dashboard said off; HTML said otherwise. Propagation, probably. But "probably" is exactly why you verify at the wire and re-check later, instead of trusting the toggle's word for it.

What actually moved the score (all free, all static)

Four changes, one commit, no plan upgrade:

1. /llms.txt — a plain-text index for agents: what the site is, every post with a one-line description, and where the machine-readable interfaces live. Generated at build time from the post collection, so it can't go stale.
2. Raw markdown twins — every post now also exists as /posts/<slug>.md, the literal source file. An agent that wants the content shouldn't have to parse my HTML to get it.
3. Link response headers — RFC 8288 pointers on every page to llms.txt, the sitemap, the JSON feed, and the API catalog. On Cloudflare Pages this is three lines in _headers.
4. /.well-known/api-catalog — an RFC 9727 linkset declaring the JSON feed as the site's machine interface. A static JSON file plus one content-type header.

And the robots.txt itself now says what I actually mean: explicit Allow for the major AI crawlers, Content-Signal: search=yes, ai-input=yes — and ai-train deliberately unset. Search and AI answers are why the site exists; granting training rights is a different decision, and "didn't think about it" shouldn't be recorded as an answer either way.

Re-scan: 43/100. Discoverability 3/4, Bot Access 2/2 — now passing for rules I actually wrote, pointing the direction I actually mean.

The points a static blog can't buy — and shouldn't chase

The remaining gap is mostly checks that don't apply. Markdown negotiation — serving markdown when a request says Accept: text/markdown — is a Cloudflare edge feature gated to paid plans; the API confirms it: "editable": false on free. The twins get agents the same bytes at a different URL. DNS-AID, MCP server cards, OAuth discovery, agent commerce: those describe sites with APIs and agent endpoints. This is a blog. Publishing an OAuth discovery document for a site with nothing to authenticate against wouldn't make it more agent-ready — it would make it lie in a machine-readable format.

A score is an instrument, and instruments have a declared scope. 43/100 with every applicable check green is a better state than 70/100 earned by scaffolding fake endpoints — the same way a test suite padded with assertion-free tests has a better number and a worse truth.

The checklist, if you run a static site and want agents reading it: curl your served robots.txt (not your repo's), find who else is speaking for your domain, then llms.txt, markdown twins, Link headers, api-catalog. An afternoon, all of it — and most of the afternoon is the toggle maze.