Scrape HTML

Use HTML when you need exact DOM structure, attributes, scripts, data layers or your own parser. Use Markdown when an LLM needs readable page content.

Yes. Set includeFrames when iframe content matters. The response marks frame URLs separately so your parser can decide how to use them.

The router starts with a fast fetch and escalates to browser-style engines when quality is low or the page needs rendering.

Use GET /v1/web/scrape/html. Local development accepts the X-API-Key header with mh-localhost-dev-key, and production clients can use the same shape with their own key.

Yes. The same endpoint can be called with curl, the TypeScript SDK, Python SDK, MCP server, Zapier, Make, n8n, Google Sheets or Excel workflows where appropriate.

When maxAgeMs is supported, a cached response can be reused if it is younger than the requested freshness window. Set maxAgeMs to 0 when you need a fresh scrape or extraction.

The API returns clear states such as blocked, verification_required, login_required, robots_disallowed or permission_required. It does not promise to bypass access controls.

Most integrations store html, url, final_url, meta.engine, plus source URL, status, confidence and timestamp fields when present. Keep source metadata so users can audit values later.

For local or small workflows, loop over a list of URLs or domains. For larger jobs, use crawl, batch workflow endpoints or a queue so failures and retries are tracked cleanly.

Check success, HTTP status, error.type and retryable flags. Retry timeouts and temporary network failures, but do not retry robots, login or permission errors without changing input or authorization.

The engine router starts with fast fetching and escalates when content quality is low. Browser-style rendering is used where available, while access-control barriers are reported explicitly.

Send only domains, URLs or fields your workflow needs. Use retention controls for generated artifacts and avoid storing unnecessary raw HTML, screenshots or extracted personal data.