The problem worth solving
Extract product context and verify media assets before migrating an online catalog.
A page returning HTTP 200 is not the same thing as usable content. Navigation chrome, JavaScript placeholders, consent overlays and blocked responses can all produce text that looks fetched but should never be fed blindly to an application.
A workflow that teams can actually ship
MediaHarvester starts with an accessible fetch path, records the engine trace, scores the extracted result and exposes explicit blocked or verification states. The consumer can request clean markdown for an LLM, raw HTML for auditing, or a crawl when one URL is not enough.
For this pattern the practical entrypoint is `GET /v1/web/scrape/markdown`, combined with whichever workflow endpoint or presentation layer the product requires.
Workflow map
Scenario in practice
A knowledge-base team is importing product documentation before a support assistant answers customer questions.
Start with one observable workflow, keep source URLs available during review and expand only after the result is reliably useful.
Try the capability
GET /v1/web/scrape/markdown Local API key:mh-localhost-dev-key
GET /v1/web/scrape/markdown?url=https://web-tasarimci.com&maxAgeMs=0Implementation notes
In practice, teams begin with a small page budget and maxAgeMs set deliberately. Fresh mode is useful during testing; cached mode is a better default for stable product documentation. The same response metadata makes retries observable instead of mysterious.
When the result affects a customer-facing experience, add an approval or override path rather than treating extraction output as immutable truth.
Signals to monitor
Measure usable pages, low-quality results, refresh latency and citation coverage. A successful ingestion pipeline is not the one that fetched the most URLs; it is the one whose retrieved context remains accurate and explainable.
Metrics turn a promising prototype into a maintainable product capability and make regressions easier to catch.
Try it with MediaHarvester
Open the local playground, choose the feature matching `GET /v1/web/scrape/markdown` and test with a public or authorized target.
The local project includes API, documentation, SDK, CLI, MCP and no-code surfaces so the same idea can move from exploration into implementation.
FAQ
Questions teams ask before implementing this workflow
What does this web extraction workflow return?
It uses GET /v1/web/scrape/markdown and related MediaHarvester surfaces to return structured context together with metadata appropriate to the workflow.
Can I test this locally?
Yes. Run the local service at http://127.0.0.1:8013 and send X-API-Key: mh-localhost-dev-key to protected API routes.
Does it work with private or blocked pages?
The platform is designed for publicly accessible or authorized sources. Verification, login, permission and robots restrictions are reported rather than bypassed.
Can this be automated?
The same API surfaces are available through CLI, Python and TypeScript SDKs, MCP tools and starter no-code integration templates.
How do I keep the result current?
Use cache freshness controls such as maxAgeMs where exposed, and schedule refreshes in a production worker only as frequently as the business case needs.