Overview
Crawl4AI is a web crawler built for feeding pages to language models. It turns web pages into clean Markdown or structured JSON, which is the format RAG pipelines and agents actually want. There are three ways to run it: as a Python library, as your own Docker server, or through Crawl4AI Cloud, a hosted API run by the same team. The crawler that powers the cloud service is the same Apache-2.0 code in the repository.
Key capabilities
- Page to Markdown: scrape a URL and get LLM-ready Markdown in one call.
- Structured extraction: pull JSON out of pages for data pipelines.
- Hosted search: the cloud API adds web search, SERP and answer endpoints that the library doesn't have.
- Batch jobs: scrape many URLs at once through the batch and jobs endpoints.
- MCP server: the hosted API can be added to Claude Code as an MCP tool.
Tradeoffs
- Self-hosting means you run the browsers and handle JavaScript-heavy pages, bot walls and proxies yourself.
- The hosted API handles those for you but is metered by credits.
- The library is Python; other languages use the HTTP API.
Ideal for
- RAG pipelines that ingest web content
- Agents that need to read live web pages
- Teams who want an open-source scraper they can self-host
Not ideal for
- One-off copying of a single page
- Scraping sites whose terms forbid it
Getting started
- Install the library with
pip install -U crawl4ai, then runcrawl4ai-setuponce to install the browser. - Create an
AsyncWebCrawler, callarun(url=...), and readresult.markdown. - If you'd rather not run browsers yourself, get a free Crawl4AI Cloud key and call
POST https://api.crawl4ai.com/scrapewith the URL; the same key works for/extractand/search. - To use it from Claude Code, add the hosted MCP server with
claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcpand your key as a bearer header.
Technical notes
- Install:
pip install -U crawl4ai, thencrawl4ai-setupto install the browser - Self-host: Python library or Docker server
- Hosted API base: api.crawl4ai.com
- Source: unclecode/crawl4ai
- GitHub stars: 87,258 (as of 2026-11-12)