Web Crawler MCP server for AI agents

The Web Crawler MCP Server provides a robust way to crawl websites through the Model Context Protocol (MCP). It allows you to extract content from web pages with configurable crawling parameters such as depth, delay, and concurrency settings.

Installation

Prerequisites

Node.js (v18+)
npm (v9+)

Setup Process

Clone the repository:

git clone https://github.com/jitsmaster/web-crawler-mcp.git
cd web-crawler-mcp

Install dependencies:
```
npm install
```
Build the project:
```
npm run build
```

Configuration

Create a .env file in the root directory with the following environment variables:

CRAWL_LINKS=false
MAX_DEPTH=3
REQUEST_DELAY=1000
TIMEOUT=5000
MAX_CONCURRENT=5

Configuration Options

Environment Variable	Default	Description
CRAWL_LINKS	false	Whether to follow links
MAX_DEPTH	3	Maximum crawl depth
REQUEST_DELAY	1000	Delay between requests (ms)
TIMEOUT	5000	Request timeout (ms)
MAX_CONCURRENT	5	Maximum concurrent requests

Running the Server

Start the MCP server with:

npm start

MCP Integration

Server Configuration

Add the following to your MCP settings file to integrate the web crawler:

{
  "mcpServers": {
    "web-crawler": {
      "command": "node",
      "args": ["/path/to/web-crawler/build/index.js"],
      "env": {
        "CRAWL_LINKS": "false",
        "MAX_DEPTH": "3",
        "REQUEST_DELAY": "1000",
        "TIMEOUT": "5000",
        "MAX_CONCURRENT": "5"
      }
    }
  }
}

Using the Crawler

The server provides a crawl tool that can be accessed through MCP. Here's an example of how to use it:

{
  "url": "https://example.com",
  "depth": 1
}

This will crawl the specified URL to the given depth and return the extracted content according to your configuration settings.

How to add this MCP server to Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > MCP and click "Add new global MCP server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "cursor-rules-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "cursor-rules-mcp"
            ]
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explictly ask the agent to use the tool by mentioning the tool name and describing what the function does.