home / mcp / anycrawl mcp server
AnyCrawl MCP Server, with Scrape, Crawl and SERP.
Configuration
View docs{
"mcpServers": {
"any4ai-anycrawl-mcp-server": {
"url": "https://mcp.anycrawl.dev/{API_KEY}/mcp",
"headers": {
"LOG_LEVEL": "debug",
"ANYCRAWL_HOST": "127.0.0.1",
"ANYCRAWL_PORT": "3000",
"ANYCRAWL_API_KEY": "YOUR-API-KEY",
"ANYCRAWL_BASE_URL": "https://your-api-server.com"
}
}
}
}AnyCrawl MCP Server enables you to drive web scraping and crawling workflows for large language model clients through the Model Context Protocol. It provides flexible modes, robust output formats, and scalable deployment options so you can fetch, parse, and organize web data for use in diverse AI applications.
You interact with the AnyCrawl MCP Server through an MCP client by connecting to either the cloud service or a self-hosted instance. Choose the transport that matches your workflow: a streamable HTTP MCP connection for integration with web-based or server-side clients, or a standard STDIO setup for local tooling and scripting. You will typically perform single-page extractions, multi-page crawls, or search-driven discoveries, and you will receive structured results in your preferred output format.
For common tasks, you will: 1) configure your API key and base URL if using self-hosted or cloud endpoints, 2) select the appropriate MCP transport mode (MCP, SSE, or STDIO), and 3) issue tool calls to scrape, crawl, or search. The server returns results with clear status updates, enabling you to monitor progress and handle errors gracefully.
Prerequisites you need before installation: Node.js 18+ and npm. Ensure you have a working environment for running Node-based tooling and containers if you plan to use Docker Compose.
Step-by-step installation and setup options are shown here. Follow the sequence that matches your preferred deployment method.
# Option 1: Run with npx (temporary install)
ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp
# Option 2: Install globally and run
npm install -g anycrawl-mcp-server
ANYCRAWL_API_KEY=YOUR-API-KEY anycrawl-mcp
# Optional: configure environment for cloud/self-hosted usage
export ANYCRAWL_API_KEY="your-api-key-here"
export ANYCRAWL_BASE_URL="https://your-api-server.com" # for self-hosted deploymentsConfiguration and deployment options are designed to be flexible. You can use the cloud service for a serverless experience, or self-hosted deployments for full control. Security is handled through API keys, and you can expose MCP endpoints via HTTP or SSE depending on your client needs.
Key practical notes include: using the MCP endpoints with an API key in the path or headers, choosing the correct mode for your client, and selecting the appropriate output formats for your data. You can also inspect health endpoints to verify the server is running and ready to process requests.
Troubleshooting tips include verifying API key configuration, ensuring network access to cloud endpoints or self-hosted URLs, checking logs for validation or network errors, and using the health endpoint to confirm connectivity.
Scrape a single URL with a chosen engine and output format, providing precise control over timeouts, retries, and content filtering.
Initiate a crawl of a website to retrieve multiple pages with configurable depth, limits, and scraping options, returning aggregated results.
Query the status of an ongoing crawl job to monitor progress and remaining work.
Fetch the results of a completed or in-progress crawl, with pagination support.
Cancel a currently queued or running crawl job to conserve resources.
Perform web searches using the AnyCrawl search engine and scrape results according to specified options and formats.