Scrape Markdown

Markdown keeps headings, lists, tables and links while removing most layout noise. It is easier to chunk and cheaper to place in a model context than raw HTML.

Yes. Keep includeLinks enabled when your agent needs source links or when downstream answers should cite the original page.

Yes. useMainContentOnly tells the extractor to focus on the article or page body and skip repeated chrome where possible.

Use GET /v1/web/scrape/markdown. Local development accepts the X-API-Key header with YOUR_API_KEY, and production clients can use the same shape with their own key.

Yes. The same endpoint can be called with curl, the TypeScript SDK, Python SDK, MCP server, Zapier, Make, n8n, Google Sheets or Excel workflows where appropriate.

When maxAgeMs is supported, a cached response can be reused if it is younger than the requested freshness window. Set maxAgeMs to 0 when you need a fresh scrape or extraction.

The API returns clear states such as blocked, verification_required, login_required, robots_disallowed or permission_required. It does not promise to bypass access controls.

Most integrations store markdown, url, links, images, plus source URL, status, confidence and timestamp fields when present. Keep source metadata so users can audit values later.

For local or small workflows, loop over a list of URLs or domains. For larger jobs, use crawl, batch workflow endpoints or a queue so failures and retries are tracked cleanly.

Check success, HTTP status, error.type and retryable flags. Retry timeouts and temporary network failures, but do not retry robots, login or permission errors without changing input or authorization.

The engine router starts with fast fetching and escalates when content quality is low. Browser-style rendering is used where available, while access-control barriers are reported explicitly.

Send only domains, URLs or fields your workflow needs. Use retention controls for generated artifacts and avoid storing unnecessary raw HTML, screenshots or extracted personal data.