Why this subject deserves an implementation guide
The starting question for this article came from a public Context.dev post titled "Rotating Proxies in Web Scraping". Rather than reproduce that article, this guide asks what the same product problem looks like inside MediaHarvester and what can be verified in the running application.
Extraction quality becomes visible when a developer needs either clean markdown for reasoning or raw HTML for evidence and debugging. The two outputs solve different problems.
The MediaHarvester approach
Fetch one representative page first, compare HTML and markdown output, inspect the fallback trace and only then expand into multi-page crawl or schema extraction.
The primary surface for this workflow is `GET /v1/compliance/error-codes`. It can be tried from the API playground and integrated through the local API key, SDK, CLI or MCP layer.
Workflow map
A realistic application scenario
An engineering team is adding website extraction to an enterprise product and must explain failures to security and customers.
The workflow begins with a narrowly scoped public or authorized source, records the endpoint output and makes the result reviewable before it becomes visible to users or informs an automated decision.
Try the capability
GET /v1/compliance/error-codes Local API key:mh-localhost-dev-key
GET /v1/compliance/error-codesImplementation choices that matter
A browser-rendered fallback should be used because the page needs it, not because it sounds more powerful. Quality scoring keeps the escalation decision measurable.
This matters because a production feature is judged less by a perfect demo result than by how it behaves when an asset is missing, a source changes, a response is cached or a request is not allowed.
How to measure whether it works
Watch usable-content rate, fallback frequency, median latency, source citation completeness and failures caused by blocked or empty content.
The app should retain enough source and request metadata to debug poor results while applying appropriate retention and access policies for customer data.
A responsible next step
Run the included endpoint against a website you control or are authorized to process, inspect the response in Visual and JSON modes, then decide which fields deserve automation and which deserve human approval.
MediaHarvester deliberately treats blocked, verification-required, login-required, robots-disallowed and permission-required outcomes as information, not obstacles to be bypassed.
FAQ
Questions teams ask before implementing this workflow
What does this trust and reliability workflow return?
It uses GET /v1/compliance/error-codes and related MediaHarvester surfaces to return structured context together with metadata appropriate to the workflow.
Can I test this locally?
Yes. Run the local service at http://127.0.0.1:8013 and send X-API-Key: mh-localhost-dev-key to protected API routes.
Does it work with private or blocked pages?
The platform is designed for publicly accessible or authorized sources. Verification, login, permission and robots restrictions are reported rather than bypassed.
Can this be automated?
The same API surfaces are available through CLI, Python and TypeScript SDKs, MCP tools and starter no-code integration templates.
How do I keep the result current?
Use cache freshness controls such as maxAgeMs where exposed, and schedule refreshes in a production worker only as frequently as the business case needs.