Crawl Website

Use maxPages, maxDepth and urlRegex together. That keeps the crawler focused on docs, pricing, blog or another path family instead of following every internal link.

Yes, enable followSubdomains when the job should move from a root domain to docs, help or app subdomains. Keep it off when you only want the starting host.

Small crawls can return immediately. Larger crawls should be treated as jobs so your app can poll status, store partial output and retry failed pages safely.

Use POST /v1/web/crawl. Local development accepts the X-API-Key header with YOUR_API_KEY, and production clients can use the same shape with their own key.

Yes. The same endpoint can be called with curl, the TypeScript SDK, Python SDK, MCP server, Zapier, Make, n8n, Google Sheets or Excel workflows where appropriate.

When maxAgeMs is supported, a cached response can be reused if it is younger than the requested freshness window. Set maxAgeMs to 0 when you need a fresh scrape or extraction.

The API returns clear states such as blocked, verification_required, login_required, robots_disallowed or permission_required. It does not promise to bypass access controls.

Most integrations store results[].url, results[].markdown, results[].depth, metadata.numSucceeded, plus source URL, status, confidence and timestamp fields when present. Keep source metadata so users can audit values later.

For local or small workflows, loop over a list of URLs or domains. For larger jobs, use crawl, batch workflow endpoints or a queue so failures and retries are tracked cleanly.

Check success, HTTP status, error.type and retryable flags. Retry timeouts and temporary network failures, but do not retry robots, login or permission errors without changing input or authorization.

The engine router starts with fast fetching and escalates when content quality is low. Browser-style rendering is used where available, while access-control barriers are reported explicitly.

Send only domains, URLs or fields your workflow needs. Use retention controls for generated artifacts and avoid storing unnecessary raw HTML, screenshots or extracted personal data.