AI ReadinessFeature

Make sure AI assistants can read your site

An AI assistant can only cite pages its crawler can fetch and read. AI Readiness checks your robots.txt, your CDN firewall and your HTML for the things that quietly keep those crawlers out.

Book a DemoSee Pricing

robots.txt per Crawler

Whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot are allowed.

Blocked at the Edge

With Cloudflare connected, we read firewall events to see whether a rule or bot challenge is blocking an AI crawler.

Content in the HTML

How many pages carry their content in the HTML and how many need JavaScript, which many crawlers do not run.

Why It Matters

Two places a crawler can be turned away

A site can allow a crawler in robots.txt and still block it with a firewall rule or a bot challenge, and the reverse happens too. Looking at both, along with whether the page content is readable without scripts, gives a straight answer to whether a crawler can use your pages.

  • robots.txt answer per AI crawler
  • CDN answer from recent firewall events: allowed, blocked or unknown
  • Share of pages that are client-rendered shells
  • Whether your homepage content is in the HTML
Where It Lives

Part of every crawl

AI Readiness is a section of Site Crawl, so it updates on the same schedule and after deploys. If a release adds a robots.txt rule that blocks an AI crawler, you see it with the rest of the crawl findings.

  • Runs with the weekly crawl and after deploys
  • Without a CDN connection, the edge answer shows as unknown rather than a guess
Explore

More in RankDebug

Site Check

Catches robots.txt, sitemap, host and noindex breaks within a day

Site Crawl

A polite crawler that finds page-level issues and diffs every run

Deploy Tracking

Lines up every release with the clicks before and after it

Opportunities

Near-wins, low CTR, decay and cannibalisation, ranked by clicks at stake

Agent

Ask questions about your search data and get answers with evidence

Reports

Written analysis from our team, with live charts and comments

AI Readiness: questions and answers

Do you track whether AI assistants mention or cite my site?

No. AI Readiness checks access: whether crawlers can reach and read your pages. Tracking what assistants say about you is a separate problem we have not built.

Should I allow every AI crawler?

That is your decision. Some crawlers collect training data and others fetch pages to answer a question in the moment. We show you what each one is allowed to do so the choice is deliberate.

Which CDNs are supported for the edge check?

Cloudflare today, through a read-only API token. Without it, the robots.txt and HTML checks still run.

See AI Readiness on your own site

In a short call we connect your site, show you what RankDebug finds, and agree a plan that fits it.

Book a Demo