Scrape Sitemap

Usually yes. Sitemap gives you a clean seed list and helps avoid wasting crawl budget on navigation-only pages.

Yes. Use urlRegex to keep only matching URLs, for example paths that contain /blog/, /docs/ or /products/.

The endpoint reports what it found. You can then fall back to /v1/web/crawl starting from the homepage or a known section URL.

Use GET /v1/web/scrape/sitemap. Local development accepts the X-API-Key header with YOUR_API_KEY, and production clients can use the same shape with their own key.

Yes. The same endpoint can be called with curl, the TypeScript SDK, Python SDK, MCP server, Zapier, Make, n8n, Google Sheets or Excel workflows where appropriate.

When maxAgeMs is supported, a cached response can be reused if it is younger than the requested freshness window. Set maxAgeMs to 0 when you need a fresh scrape or extraction.

The API returns clear states such as blocked, verification_required, login_required, robots_disallowed or permission_required. It does not promise to bypass access controls.

Most integrations store urls[], meta.sitemapsFetched, meta.sitemapsSkipped, meta.errors, plus source URL, status, confidence and timestamp fields when present. Keep source metadata so users can audit values later.

For local or small workflows, loop over a list of URLs or domains. For larger jobs, use crawl, batch workflow endpoints or a queue so failures and retries are tracked cleanly.

Check success, HTTP status, error.type and retryable flags. Retry timeouts and temporary network failures, but do not retry robots, login or permission errors without changing input or authorization.

The engine router starts with fast fetching and escalates when content quality is low. Browser-style rendering is used where available, while access-control barriers are reported explicitly.

Send only domains, URLs or fields your workflow needs. Use retention controls for generated artifacts and avoid storing unnecessary raw HTML, screenshots or extracted personal data.