Site Crawls

Read crawls of the workspace website, their findings, and every page of one finding.

These endpoints need the search scope. Starting a crawl and changing crawler access are done in the dashboard. The Site Crawl guide explains how crawls run.

A crawl

Every crawl in a response has this shape:

Field	Notes
`id`	The crawl id
`trigger`	`deploy`, `scheduled` or `manual`
`status`	`queued`, `running`, `done` or `failed`
`pageCap`	The page budget
`pagesCrawled`, `pagesDiscovered`	Pages fetched, and URLs found
`pagesSampledOut`	URLs skipped because their URL pattern reached its sample size
`complete`	`true` when the crawl reached every page it found within the budget
`blockedPages`, `rateLimited`	How many fetches were refused, and whether the site rate-limited the crawl
`clientRenderedPages`, `renderedPages`	Pages that arrived as client-side shells, and how many were rendered in a browser
`sitemapUrls`, `edgeCount`	URLs in the sitemaps, and internal links found
`aiReadiness`	Per AI crawler, robots.txt and edge access, and how much content is in the HTML. See AI readiness
`error`	Why a crawl failed or stopped early
`createdAt`, `startedAt`, `finishedAt`	Timestamps

A finding in a crawl report has kind, severity, count (pages it names in total), urls (the first pages), details (a short detail per URL) and clicks (Search Console clicks on those pages over the last 28 days).

Latest crawl

GET /v1/site-crawls/latest?workspace_id={id}

latest is the newest crawl in any state. crawl is the newest finished one, with its findings, and baseline is the finished crawl before it. delta lists the findings added and resolved between the two, or is null when there is nothing to compare.

List crawls

GET /v1/site-crawls?workspace_id={id}

Crawl history, newest first. limit is 1 to 100, default 20.

Get a crawl

GET /v1/site-crawls/{id}

One crawl with its findings and the delta against the crawl before it, in the same shape as the latest crawl.

Every page of one finding

GET /v1/site-crawls/{id}/findings/{kind}

kind is a crawl finding kind such as broken_page; the findings reference lists them all. The response has the finding's kind, severity, count and clicks, and urls with a url, detail and clicks for each page.

Download a finding as CSV

GET /v1/site-crawls/{id}/findings/{kind}/urls.csv

The same pages as a CSV file, with the URL, the detail and the 28-day clicks per page.

On this page