superbrickstudio/website-ai-readiness

Website AI-readiness check

Check a website the way AI search engines and crawlers see it, then explain what to fix in plain English, most important first.

You need to be able to fetch URLs (curl in a shell, or a web fetch tool). Keep it polite: a handful of requests per site, never a crawl.

What to ask for

Only the website address. If the user names specific pages (a pricing page, a blog post), check those as well as the home page. For a migration, ask for the list of old URLs.

The checks

Run these in order. Record what you find for the report at the end.

1. robots.txt: which crawlers are allowed

Fetch https://<domain>/robots.txt.

• No file (404): crawlers treat that as everything allowed. Fine, but recommend one with a Sitemap: line.
• An HTML page instead of a text file: the site has no real robots.txt. Same as missing.

Read it the way Google and the AI companies document it:

• Rules come in groups: one or more User-agent: lines, then Allow: and Disallow: lines.
• A crawler follows the group that names it. If none names it, it follows the User-agent: * group. If there's neither, everything is allowed.
• Within that group, the longest matching path wins. If an Allow and a Disallow are the same length, Allow wins.
• An empty Disallow: blocks nothing. Disallow: / blocks the whole site.

Give a verdict for each crawler below: allowed, partly blocked (ordinary content pages such as /about or /blog are blocked, not just /admin or /cart) or blocked (the home page is blocked).

Token	Company	What blocking it does
OAI-SearchBot	OpenAI	Keeps the site out of ChatGPT search results
ChatGPT-User	OpenAI	Stops ChatGPT opening pages when a user asks
Claude-SearchBot	Anthropic	Keeps the site out of Claude's search
Claude-User	Anthropic	Stops Claude opening pages when a user asks
PerplexityBot	Perplexity	Keeps the site out of Perplexity answers
Perplexity-User	Perplexity	Stops Perplexity opening pages when a user asks
DuckAssistBot	DuckDuckGo	Keeps the site out of DuckDuckGo's AI answers
GPTBot	OpenAI	Opts out of training OpenAI's models (not search)
ClaudeBot	Anthropic	Opts out of training Claude (not search)
Google-Extended	Google	Opts out of Gemini training; doesn't affect Google Search or AI Overviews
Applebot-Extended	Apple	Opts out of Apple's AI training
meta-externalagent	Meta	Opts out of Meta's AI
CCBot	Common Crawl	Opts out of the open dataset many models train on
Bytespider	ByteDance	Opts out of ByteDance's AI
Googlebot	Google	Google Search, and AI Overviews
Bingbot	Microsoft	Bing and Copilot, and partly ChatGPT search
Applebot	Apple	Siri and Spotlight suggestions

The distinction that matters most: search crawlers (the first seven, plus Googlebot and Bingbot) decide whether the site can appear in AI answers. Training crawlers only decide whether its content trains future models. Blocking training while allowing search is a common, reasonable choice. Blocking search crawlers is usually a mistake, often inherited from a template or a security plugin.

Also flag:

• No Sitemap: line.
• Lines that aren't valid robots.txt directives (typos such as Dissallow).

2. Firewall and bot blocking

robots.txt can allow a crawler while the server or CDN turns it away anyway (Cloudflare's AI bot blocking is the common one).

Fetch the home page twice:

# As a browser
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36" https://example.com/

# As an AI search crawler
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot" https://example.com/

Repeat the second request with Claude-SearchBot/1.0; +https://www.anthropic.com and PerplexityBot/1.0; +https://perplexity.ai/perplexitybot in place of the OpenAI token.

If the browser request succeeds but a crawler gets 401, 403, 429 or 503, or a challenge page ("Just a moment...", "Attention required", cf-chl), the firewall is blocking that crawler. If the browser request is also turned away, say you couldn't test it, rather than guessing.

Real crawlers come from published IP ranges, so a server may treat your request differently from the real thing. Present a block as "likely", and suggest checking the firewall or CDN settings.

3. Can the content be read without JavaScript?

Most AI crawlers read the HTML as served and don't run JavaScript. Fetch the home page and one or two key pages, and check the raw HTML (not a rendered browser view) for:

• A <title> and a meta description.
• One <h1> that says what the page is about.
• The main body text: the headline, the key paragraphs, prices or product details.

If the HTML is mostly an empty <div id="root"> or <div id="app"> plus scripts, the content only appears after JavaScript runs. AI crawlers then see little or nothing. Recommend server-side rendering or pre-rendering.

Also check the page doesn't carry <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header by accident.

4. Structured data

Extract every <script type="application/ld+json"> block from the raw HTML. For each item, note its @type and which of these expected properties are missing:

Type	Expected properties
Organization	name, url, logo, sameAs
LocalBusiness	name, address, telephone, url, openingHoursSpecification, geo
ProfessionalService	name, address, telephone, url
WebSite	name, url
WebPage	name, url
FAQPage	mainEntity
Article, BlogPosting, NewsArticle	headline, author, datePublished, image
Product	name, image, offers, aggregateRating
Service	name, provider, areaServed, description
BreadcrumbList	itemListElement
Person	name, jobTitle, url, sameAs
Event	name, startDate, location

Flag:

• JSON-LD that doesn't parse (a syntax error means none of it counts).
• No Organization (or LocalBusiness) on the home page. It's the clearest statement of who the business is.
• Schema added by JavaScript only. If a rendered view shows schema the raw HTML doesn't, most AI crawlers won't see it.

5. Redirects

Check the variants of the home page, and any old URLs the user gives you:

curl -sIL -o /dev/null -w "%{num_redirects} redirects -> %{http_code} %{url_effective}\n" http://example.com/

Check http://, https://, with and without www. To see each hop, run curl -sI and follow each Location header by hand.

Verdicts:

• OK: no redirect, or one permanent redirect (301 or 308) to the final page.
• Chain: more than one redirect in a row. Point the first address straight at the last.
• Temporary: a 302 or 307 for a page that has moved for good. Use 301 or 308.
• Loop: the redirects go round in a circle, so the page never loads.
• Too many: more than 10 hops. Google stops following after 10.
• Broken: it ends on a 4xx or 5xx.
• To the home page: an old page redirected to the home page instead of its closest match.
• Meta refresh: a redirect done in the page rather than by the server.

6. Sitemap

Find the sitemap from robots.txt, or try /sitemap.xml and /sitemap_index.xml. Check:

• It loads, and is XML.
• It lists the site's real pages, using the final URLs (no redirects, no 404s).
• It has <lastmod> dates, and they look real rather than all the same day.

7. llms.txt

Fetch https://<domain>/llms.txt.

• If it exists, check it follows the format at llmstxt.org: an # H1 with the site's name, a > one-line summary, then ## sections of - [Title](URL): note links. Check that the links work.
• If it doesn't, offer to draft one. Only include URLs you found in the sitemap or on the site. Never guess a URL. Group the key pages (services or products, pricing, about, contact, the best guides) under clear headings, and put the rest under ## Optional.

Be honest about what it does: llms.txt is a proposed standard. It gives AI tools a clean map of the site, but no major AI search engine has said it uses the file for ranking.

The report

Lead with a one-line verdict: can AI search find and read this site, yes, mostly, or no?

Then list the findings as a short table or list, in this order:

1. Blocks AI search entirely: search crawlers blocked in robots.txt or by the firewall, noindex by accident, content that only appears with JavaScript.
2. Hurts how the site is understood: missing or broken structured data, missing titles or descriptions, broken redirects.
3. Good housekeeping: no sitemap line, stale lastmod dates, no llms.txt, training-crawler choices to review.

For each finding, give:

• What you found, quoting the actual line or status code.
• Why it matters, in one plain sentence.
• The fix: the exact robots.txt line, header or setting to change.

Don't pad the report. If a check passed, one line saying so is enough. Never report a problem you didn't observe.

Free tools and further reading

If the user wants a visual report or a second opinion, Superbrick's free checkers run the same checks in a browser:

• robots.txt and AI crawler checker: https://superbrick.studio/robots-txt-checker
• Redirect checker, up to 50 URLs at once: https://superbrick.studio/redirect-checker
• Schema markup generator and checker: https://superbrick.studio/schema-markup-generator
• llms.txt generator: https://superbrick.studio/llms-txt-generator
• A full AI visibility (AEO) assessment: https://superbrick.studio/aeo-assessment

Guides:

• Writing pages AI assistants can quote: https://superbrick.studio/journal/writing-pages-ai-assistants-can-quote
• AEO vs GEO vs SEO: https://superbrick.studio/journal/aeo-vs-geo-vs-seo
• Website migration SEO: https://superbrick.studio/journal/website-migration-seo