crawl_site
Crawl a project's website (async). Returns a job_id immediately; poll it with get_job_status.
Cost: 20 credits.
crawl_site crawls the website behind one of your projects — following its sitemap and same-origin links, respecting robots.txt — and records the pages for later audits. It is asynchronous: the call returns a job_id immediately instead of waiting for the crawl to finish, so the MCP request never times out on a large site. The crawl is charged only when it runs — a crawl that reaches no pages is not charged.
What it does
Queues a crawl for the project's domain and hands you a job_id. A background worker runs the crawl and stores the result; you check progress with get_job_status.
Large sites
Each crawl covers up to 100 pages. To crawl a bigger site, target a section with include_paths — for example ["/blog"] — and run one focused crawl per section; this keeps every crawl within the cap and spends predictably.
Before queuing, crawl_site runs a quick, free size check. If your site is very large, it first returns a confirmation — nothing is charged — that states this run's flat cost and, kept separate, an informational projection of what crawling the whole site would take at the current rate. The projection is never what you are charged; it just means a big site can't silently run up cost. Re-run with "confirm": true to proceed, or narrow the scope with include_paths.
Example
Ask your MCP client in plain language:
Crawl my example.com project.
The tool replies with a job_id. Poll it until the job is done:
What's the status of job
<job_id>?
Input
| Field | Type | Required | Description |
|---|---|---|---|
project_id | string (uuid) | Yes | The project_id from setup_project / list_projects. |
max_urls | integer | No | Maximum pages to crawl (1–100, default 100). |
include_paths | string[] | No | Limit the crawl to URL paths starting with these prefixes, e.g. ["/blog"]. Omit to crawl the whole site (up to the page cap). |
Returns
A job_id, a status of queued, and the estimated_credits the crawl will cost. Feed the job_id to get_job_status to watch it finish and read the summary.