RAG

RAG Website Ingestion

Convert websites into clean markdown, metadata, links and source-aware chunks for retrieval-augmented generation.

Sitemap Markdown Depth Chunks

Ingest a website Read docs

01 Markdown crawl

02 Sitemap seeding

03 Depth and page limits

04 Cache freshness controls

How it works

From input to useful output in three steps

POST /v1/web/crawl

Seed

Start from sitemap or URL

Use sitemap discovery when available, then crawl with page and depth limits.

GET /v1/web/scrape/sitemap

Clean

Convert pages to markdown

Keep main content, links and metadata so downstream chunks remain explainable.

GET /v1/web/scrape/markdown

Index

Store with source metadata

Embed chunks with URL, title, crawl depth and refresh timestamp.

url + markdown + metadata

Example workflow

Website to retrieval-ready corpus

Knowledge base needs

Fresh public docs with source URLs and clean text.

MediaHarvester crawls

Sitemap URLs, markdown pages, titles, links and page status.

RAG pipeline stores

Source-aware documents ready for embedding and retrieval.

{
  "url": "https://help.example.com",
  "maxPages": 50,
  "maxDepth": 3
}

Built for

Teams that can use this immediately

AI knowledge bases

Challenge: Docs change after ingestion.

Solution: Refresh selected pages on schedule.

Impact: Answers stay current.

Support search

Challenge: Search results lack source context.

Solution: Store titles and URLs with markdown.

Impact: Users can verify answers.

Developer tools

Challenge: Agents need docs inside the IDE.

Solution: Expose crawl through SDK or MCP.

Impact: Agents work with current docs.

Implementation FAQ

Questions teams ask before building RAG Website Ingestion

12 answers

What does RAG Website Ingestion help me build?

It turns Website ingestion API for RAG and knowledge base pipelines. into a repeatable workflow. The first practical outcome is markdown crawl.

Which API should I call first?

Start with POST /v1/web/crawl. The workflow section shows the supporting endpoints to add when you need more context, evidence or visual assets.

What input do I need from my user?

The smallest useful input is shown in the example workflow: Fresh public docs with source URLs and clean text. Ask only for the domain, URL, email, ticker or descriptor that the selected endpoint actually needs.

What should I store from the response?

Store the normalized result together with source URLs, confidence or access state, and the time it was fetched. This keeps later refreshes and human review explainable.

Can I use this from a backend, CLI or AI agent?

Yes. The REST endpoint, TypeScript and Python SDKs, CLI and MCP tools use the same local API contract. No-code templates are available for business workflows.

How do I keep the result current?

Use maxAgeMs where the endpoint supports it, or schedule the relevant crawl or workflow. Refresh facts that can change and cache stable assets such as logos more aggressively.

What happens when the website blocks access?

The API reports blocked, verification_required, login_required, robots_disallowed or permission_required instead of treating an empty page as a successful result.

How should I handle missing fields?

Keep the UI usable with optional fields, sensible placeholders and manual overrides. A missing logo, address or classification should not prevent the rest of the workflow from completing.

Who is this workflow designed for?

The examples focus on ai knowledge bases, support search, developer tools, but the API response is general enough to use in custom applications.

Can I review the result before publishing it?

Yes. Show extracted values, source URLs and generated visual previews in an approval step before changing customer-facing content or records.

Is batch processing supported?

Use the batch, crawl or workflow endpoint shown for the capability when available. For smaller integrations, process a controlled list and record individual failures rather than failing the whole job.

How do I try the complete example locally?

Open the linked playground from this page or call POST /v1/web/crawl at http://127.0.0.1:8013 with X-API-Key: mh-localhost-dev-key. The example payload on this page is ready to adapt.

Related use cases

Explore adjacent workflows

Power LLMs with Web Context Give AI agents real-time website context with clean markdown, crawl results, screenshots, brand data and structured extraction. Reduce Onboarding Friction Personalize onboarding automatically with company logos, colors, fonts, styleguides, descriptions and industry classification. Enrich Company Profiles Build richer CRM, directory and marketplace profiles with logos, descriptions, socials, colors, addresses and industry data. Automated Brand Kits Generate brand kits on demand with logos, colors, typography, component tokens, CSS variables and Tailwind theme exports. Stock Ticker Enrichment Resolve public-company tickers and ISINs into company websites, logos, brand profiles and stock metadata. Logo Link API Serve company logos through a spreadsheet-friendly public Logo Link endpoint with monogram fallback.