Crawl4AI RAG MCP server for AI agents

This Docker-based MCP server provides web crawling, search capabilities, and RAG functionality for AI agents and coding assistants. It combines Model Context Protocol, Crawl4AI, SearXNG, and Supabase into a complete solution that can be deployed with a single command.

Installation

To get started with the MCP server:

Clone the repository:

git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
cd mcp-crawl4ai-rag

Configure environment:

cp .env.example .env
# Edit .env with your API keys

Deploy the complete stack:
```
docker compose up -d
```

Your MCP server will be running at http://localhost:8051, with SearXNG search available internally.

Database Setup

Before using the server, you need to set up the Supabase database:

Go to the SQL Editor in your Supabase dashboard
Create a new query and paste the contents of crawled_pages.sql
Run the query to create the necessary tables and functions

Configuration

Edit your .env file to configure the server:

# Server configuration
TRANSPORT=sse
HOST=0.0.0.0
PORT=8051

# SearXNG configuration
SEARXNG_URL=http://searxng:8080
SEARXNG_USER_AGENT=MCP-Crawl4AI-RAG-Server/1.0
SEARXNG_DEFAULT_ENGINES=google,bing,duckduckgo
SEARXNG_TIMEOUT=30
SEARXNG_HOSTNAME=http://localhost

# API keys and services
OPENAI_API_KEY=your_openai_api_key
MODEL_CHOICE=gpt-4.1-nano-2025-04-14
SUPABASE_URL=your_supabase_project_url
SUPABASE_SERVICE_KEY=your_supabase_service_key

# RAG enhancement strategies
USE_CONTEXTUAL_EMBEDDINGS=false
USE_HYBRID_SEARCH=false
USE_AGENTIC_RAG=false
USE_RERANKING=false
USE_KNOWLEDGE_GRAPH=false

For knowledge graph functionality, you'll need additional Neo4j configuration:

NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_neo4j_password

Available Tools

The MCP server provides several tools for web search and content retrieval:

Core Tools

scrape_urls: Scrape one or more URLs and store content in the vector database
smart_crawl_url: Intelligently crawl a website based on URL type
get_available_sources: Get a list of all available sources in the database
perform_rag_query: Search for relevant content using semantic search
search: Comprehensive web search that integrates SearXNG, scraping, and RAG processing

Conditional Tools

These tools are available when certain features are enabled:

search_code_examples: Available when USE_AGENTIC_RAG=true
parse_github_repository, check_ai_script_hallucinations, and query_knowledge_graph: Available when USE_KNOWLEDGE_GRAPH=true

RAG Enhancement Strategies

The server supports several RAG enhancement strategies:

Contextual Embeddings: Enhances chunk embeddings with document context
Hybrid Search: Combines vector and keyword search for better results
Agentic RAG: Extracts and indexes code examples with summaries
Reranking: Improves result relevance using cross-encoder models
Knowledge Graph: Enables repository analysis and hallucination detection

Enable these features by setting the corresponding variables to true in your .env file.

Integration with MCP Clients

After starting the Docker stack, connect to your MCP server:

Claude Desktop/Windsurf:

{
  "mcpServers": {
    "crawl4ai-rag": {
      "transport": "sse",
      "url": "http://localhost:8051/sse"
    }
  }
}

Claude Code CLI:

claude mcp add-json crawl4ai-rag '{"type":"http","url":"http://localhost:8051/sse"}' --scope user

Managing the Server

Control the Docker stack with these commands:

# Start all services
docker compose up -d

# View logs
docker compose logs -f

# Stop all services
docker compose down

# Restart specific service
docker compose restart mcp-crawl4ai

Troubleshooting

If you encounter issues:

Check logs with docker compose logs mcp-crawl4ai
Verify configuration with docker compose config
Test API connectivity within the container
For Neo4j connection issues, use host.docker.internal:7687 instead of localhost:7687
Reset everything with docker compose down -v && docker compose up -d

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "crawl4ai-rag" '{"transport":"sse","url":"http://localhost:8051/sse"}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "crawl4ai-rag": {
            "transport": "sse",
            "url": "http://localhost:8051/sse"
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "crawl4ai-rag": {
            "transport": "sse",
            "url": "http://localhost:8051/sse"
        }
    }
}

3. Restart Claude Desktop for the changes to take effect