AI Agents

Power LLMs with Web Context

Give AI agents real-time website context with clean markdown, crawl results, screenshots, brand data and structured extraction.

Crawl Markdown Sources Agent context

Build an AI web context workflow Read docs

01 Fresh markdown from public or authorized pages

02 Clear access and content status

03 Evidence URLs for extracted facts

04 Refresh controls for current results

How it works

From input to useful output in three steps

POST /v1/web/crawl

Request

Choose the web scope

Start with a single public URL or a domain. Limit depth and page count so the agent receives the pages that matter instead of the entire web.

POST /v1/web/crawl

Context

Convert pages to clean context

Each page becomes markdown with URL, title, links and status metadata. Screenshots or images can be added when visual evidence matters.

GET /v1/web/scrape/markdown

Answer

Ask the model with evidence

Send the markdown and source URLs to your LLM so the final response can cite the pages it used.

sources[] + markdown

Example workflow

Public pages to grounded answers

User asks

What changed in this product documentation and which pages support the answer?

MediaHarvester collects

Sitemap URLs, clean markdown, titles, links, screenshots and per-page status.

Your AI returns

A sourced answer with links back to the exact pages used.

{
  "url": "https://docs.example.com",
  "maxPages": 20,
  "maxDepth": 2,
  "output": "markdown"
}

Built for

Teams that can use this immediately

AI support agents

Challenge: Answers go stale when docs change.

Solution: Refresh website context before answering.

Impact: Fewer unsupported answers and clearer citations.

RAG products

Challenge: Ingestion pipelines lose page metadata.

Solution: Store markdown with URL, title and depth.

Impact: Search results become easier to audit.

Research assistants

Challenge: Manual browsing slows down every report.

Solution: Crawl selected public pages and summarize them.

Impact: Research starts with usable source material.

Implementation FAQ

Questions teams ask before building Power LLMs with Web Context

12 answers

What does Power LLMs with Web Context help me build?

It turns LLM web context API for agents, assistants and RAG pipelines. into a repeatable workflow. The first practical outcome is fresh markdown from public or authorized pages.

Which API should I call first?

Start with POST /v1/web/crawl. The workflow section shows the supporting endpoints to add when you need more context, evidence or visual assets.

What input do I need from my user?

The smallest useful input is shown in the example workflow: What changed in this product documentation and which pages support the answer? Ask only for the domain, URL, email, ticker or descriptor that the selected endpoint actually needs.

What should I store from the response?

Store the normalized result together with source URLs, confidence or access state, and the time it was fetched. This keeps later refreshes and human review explainable.

Can I use this from a backend, CLI or AI agent?

Yes. The REST endpoint, TypeScript and Python SDKs, CLI and MCP tools use the same local API contract. No-code templates are available for business workflows.

How do I keep the result current?

Use maxAgeMs where the endpoint supports it, or schedule the relevant crawl or workflow. Refresh facts that can change and cache stable assets such as logos more aggressively.

What happens when the website blocks access?

The API reports blocked, verification_required, login_required, robots_disallowed or permission_required instead of treating an empty page as a successful result.

How should I handle missing fields?

Keep the UI usable with optional fields, sensible placeholders and manual overrides. A missing logo, address or classification should not prevent the rest of the workflow from completing.

Who is this workflow designed for?

The examples focus on ai support agents, rag products, research assistants, but the API response is general enough to use in custom applications.

Can I review the result before publishing it?

Yes. Show extracted values, source URLs and generated visual previews in an approval step before changing customer-facing content or records.

Is batch processing supported?

Use the batch, crawl or workflow endpoint shown for the capability when available. For smaller integrations, process a controlled list and record individual failures rather than failing the whole job.

How do I try the complete example locally?

Open the linked playground from this page or call POST /v1/web/crawl at http://127.0.0.1:8013 with X-API-Key: mh-localhost-dev-key. The example payload on this page is ready to adapt.

Related use cases

Explore adjacent workflows

Reduce Onboarding Friction Personalize onboarding automatically with company logos, colors, fonts, styleguides, descriptions and industry classification. Enrich Company Profiles Build richer CRM, directory and marketplace profiles with logos, descriptions, socials, colors, addresses and industry data. Automated Brand Kits Generate brand kits on demand with logos, colors, typography, component tokens, CSS variables and Tailwind theme exports. Stock Ticker Enrichment Resolve public-company tickers and ISINs into company websites, logos, brand profiles and stock metadata. Logo Link API Serve company logos through a spreadsheet-friendly public Logo Link endpoint with monogram fallback. Power Generative AI Enhance AI generation with real-time brand context, product facts, pricing pages, images and styleguide tokens.