AI Readiness

AI assistants that answer with links to websites first need their crawlers to reach those sites. AI readiness checks the technical side of that: whether each AI crawler is allowed by robots.txt, whether the edge lets it through, and whether your content is in the HTML it receives. It does not measure what any assistant says about your site.

The check runs with every site crawl and appears as AI Readiness on the Crawl page.

Crawlers

For each of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot, two columns:

• Robots.txt: whether your robots.txt allows that crawler on the homepage. It reads Unknown when robots.txt could not be read.
• Edge: whether your CDN blocked or challenged that crawler in the last 28 days. This needs the Cloudflare connection, which supplies the firewall events. A crawler with any blocking or challenge action against it in that time reads Blocked, one with none reads Allowed, Without a Cloudflare connection every crawler reads Unknown, and so does a crawler the edge data cannot tell apart from other bots.

A crawler can be allowed by robots.txt and still be stopped at the edge before robots.txt is ever read, which is why both are shown.

Content in the HTML

Many AI crawlers do not run JavaScript. The check counts the crawled HTML pages whose main content is already in the HTML, against those that arrive as a client-side shell, and says whether the homepage's content is in its HTML. If many pages are shells, rendering them on the server is what makes them readable to these crawlers. See client-side rendered pages.