Web Content Explorer MCP server

The Docs Fetch MCP Server enables large language models to retrieve and explore web content autonomously. This specialized server can fetch clean web content, recursively explore linked pages, and provide comprehensive information about specific topics by traversing documentation.

Installation

Setting Up the Server

1. Clone the repository:

   git clone https://github.com/wolfyy970/docs-fetch-mcp.git
   cd docs-fetch-mcp
   

2. Install dependencies:

   npm install
   

3. Build the project:

   npm run build
   

Configuring Your Claude Client

To use the MCP server with Claude, add the following configuration to your Claude Client settings:

{
  "mcpServers": {
    "docs-fetch": {
      "command": "node",
      "args": [
        "/path/to/docs-fetch-mcp/build/index.js"
      ],
      "env": {
        "MCP_TRANSPORT": "pipe"
      }
    }
  }
}

Be sure to replace /path/to/docs-fetch-mcp with the actual path where you cloned the repository.

Using the MCP Server

Available Tool

The server provides a single MCP tool called fetch_doc_content that retrieves web content with recursive exploration capabilities.

Parameters

• url (string, required): The URL of the web page you want to fetch
• depth (number, optional): Controls how deep the server should explore linked pages
  • Default: 1
  • Range: 1-5 (higher values explore more links but take longer)

Example Usage

When invoking the tool through an MCP-enabled Claude model, use the following format:

I need to learn about [topic]. Please use fetch_doc_content to explore the documentation at [url].

Response Format

The server returns a structured JSON response with the following information:

{
  "rootUrl": "https://example.com/docs",
  "explorationDepth": 2,
  "pagesExplored": 5,
  "content": [
    {
      "url": "https://example.com/docs",
      "title": "Documentation",
      "content": "Main page content...",
      "links": [
        {
          "url": "https://example.com/docs/topic1",
          "text": "Topic 1"
        }
      ]
    }
  ]
}

Key Features

Content Capabilities

• Clean Content Extraction: Removes ads, navigation elements, and other distractions
• Link Analysis: Identifies relevant links within the content
• Same-Domain Focus: Only explores pages within the same domain as the original URL
• Smart Filtering: Prioritizes content-rich pages over navigation elements

Technical Implementation

• Uses a dual-strategy approach with fast axios requests first, then puppeteer as a fallback
• Implements parallel processing for efficient crawling
• Handles errors gracefully, returning partial results when possible
• Includes timeout prevention to ensure reliable operation

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "docs-fetch" '{"command":"node","args":["/path/to/docs-fetch-mcp/build/index.js"],"env":{"MCP_TRANSPORT":"pipe"}}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "docs-fetch": {
            "command": "node",
            "args": [
                "/path/to/docs-fetch-mcp/build/index.js"
            ],
            "env": {
                "MCP_TRANSPORT": "pipe"
            }
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "docs-fetch": {
            "command": "node",
            "args": [
                "/path/to/docs-fetch-mcp/build/index.js"
            ],
            "env": {
                "MCP_TRANSPORT": "pipe"
            }
        }
    }
}

3. Restart Claude Desktop for the changes to take effect