Apify RAG Web Browser MCP server

This MCP server acts as a bridge between AI models and the web, enabling them to search for information and extract content from websites. It integrates with the RAG Web Browser Actor on Apify to provide AI assistants with web browsing capabilities similar to ChatGPT's web search feature.

Installation

Prerequisites

Before installing the MCP server, ensure you have:

• MacOS or Windows
• Latest version of Claude Desktop (or another MCP client)
• Node.js v18 or higher
• An Apify API Token

Setup Instructions

1. Clone the repository:

git clone [email protected]:apify/mcp-server-rag-web-browser.git

2. Navigate to the project directory and install dependencies:

cd mcp-server-rag-web-browser
npm install

3. Build the project:

npm run build

Configuration

Setting Up Claude Desktop

1. Edit your Claude Desktop configuration file:

   • MacOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json
   • Windows: %APPDATA%/Claude/claude_desktop_config.json

2. Add the following configuration to the file:

"mcpServers": {
  "rag-web-browser": {
    "command": "npx",
    "args": [
      "@apify/mcp-server-rag-web-browser"
    ],
    "env": {
       "APIFY_TOKEN": "your-apify-api-token"
    }
  }
}

3. Restart Claude Desktop completely
4. Look for the 🔌 icon to confirm that the server is connected

Usage

Using with Claude

You can ask Claude to perform web searches with prompts like:

• "What is an MCP server and how can it be used?"
• "What is an LLM, and what are the recent news updates?"
• "Find and analyze recent research papers about LLMs."

Server Capabilities

The MCP server provides the following tool:

• search: Performs web searches and extracts content from websites
  • Parameters:
    • query (string, required): Search term or URL
    • maxResults (number, optional): Maximum search results to scrape (default: 1)
    • scrapingTool (string, optional): Tool for extracting web pages ('browser-playwright' or 'raw-http', default: 'raw-http')
    • outputFormats (array, optional): Output formats ('text', 'markdown', 'html', default: ['markdown'])
    • requestTimeoutSecs (number, optional): Maximum request time in seconds (default: 40)

Debugging

You can debug the server using the MCP Inspector:

export APIFY_TOKEN=your-apify-api-token
npx @modelcontextprotocol/inspector npx -y @apify/mcp-server-rag-web-browser

Local Testing

To test the server locally without Claude:

export APIFY_TOKEN=your-apify-api-token
node dist/example_client_stdio.js

This script starts the MCP server, fetches available tools, and calls the search tool with a query.

How to add this MCP server to Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > MCP and click "Add new global MCP server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "cursor-rules-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "cursor-rules-mcp"
            ]
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explictly ask the agent to use the tool by mentioning the tool name and describing what the function does.