Document Crawler & Search MCP server

This MCP server enables you to crawl websites, generate searchable documentation, and query it using Model Context Protocol tools. It's designed to integrate with clients like Cursor for enhanced AI documentation search capabilities.

Installation

Follow these steps to set up the MCP server:

Prerequisites

Install uv for dependency management by following instructions on the uv website.

Setup

1. Clone the repository:

git clone https://github.com/alizdavoodi/MCPDocSearch.git
cd MCPDocSearch

2. Install dependencies:

uv sync

This creates a virtual environment and installs all required dependencies.

Usage Guide

Crawling Documentation

The crawler converts websites into Markdown files that the MCP server can process.

Basic Usage

uv run python crawl.py https://docs.example.com

This crawls the provided URL with default settings and saves the output to ./storage/docs.example.com.md.

Advanced Options

For more control over the crawling process:

uv run python crawl.py https://docs.another.site --output ./storage/custom_name.md --max-depth 2 --keyword "API" --exclude-pattern "*blog*"

Available Options

View all available options:

uv run python crawl.py --help

Key options include:

• --output/-o: Specify output file path
• --max-depth/-d: Set crawl depth (1-5)
• --include-pattern/--exclude-pattern: Filter URLs to crawl
• --keyword/-k: Add keywords for relevance scoring
• --remove-links/--keep-links: Control HTML cleaning
• --wait-for: Wait for a specific time or CSS selector
• --page-load-timeout: Set maximum page load time in seconds

Refining Your Crawls

For more precise documentation collection:

1. Use --include-pattern to restrict which URLs are crawled (supports wildcards)
2. Set appropriate --max-depth based on how deep the documentation is nested
3. Enable verbose mode (-v) to see which URLs are being visited or skipped

Example for targeting specific documentation sections:

uv run python crawl.py https://pulsar.apache.org/docs/4.0.x/admin-api-overview/ -v --include-pattern "*admin-api*" --max-depth 2

Running the MCP Server

The MCP server is designed to be run by MCP clients like Cursor. When run, it loads Markdown files from ./storage/, processes them into semantic chunks, and makes them searchable.

Note: The first time the server runs after new documents are crawled, it may take several minutes to generate embeddings. Subsequent startups will be faster due to caching.

Configuring with Cursor

1. Create a .cursor/mcp.json file in the project root with this content:

{
  "mcpServers": {
    "doc-query-server": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/your/MCPDocSearch",
        "run",
        "python",
        "-m",
        "mcp_server.main"
      ],
      "env": {}
    }
  }
}

Important: Replace /path/to/your/MCPDocSearch with the absolute path to your project directory.

2. Restart Cursor
3. Access the server via Cursor's Agent using: @doc-query-server search documentation for "query"

Configuring with Claude for Desktop

For Claude for Desktop, follow the official documentation to set up the MCP server.

Available MCP Tools

The server exposes several tools for interacting with your documentation:

• list_documents: Lists all available crawled documents
• get_document_headings: Retrieves heading structure for a document
• search_documentation: Performs semantic search over document chunks

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "doc-query-server" '{"command":"uv","args":["--directory","/path/to/your/MCPDocSearch","run","python","-m","mcp_server.main"],"env":[]}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "doc-query-server": {
            "command": "uv",
            "args": [
                "--directory",
                "/path/to/your/MCPDocSearch",
                "run",
                "python",
                "-m",
                "mcp_server.main"
            ],
            "env": []
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "doc-query-server": {
            "command": "uv",
            "args": [
                "--directory",
                "/path/to/your/MCPDocSearch",
                "run",
                "python",
                "-m",
                "mcp_server.main"
            ],
            "env": []
        }
    }
}

3. Restart Claude Desktop for the changes to take effect