Puppeteer Vision Web Scraper MCP server for AI agents

This MCP server provides a powerful tool for scraping webpages and converting them to markdown format, handling interactive elements automatically using AI capabilities. It's designed to work with the Model Context Protocol to integrate with LLM orchestrators.

Installation

To use the Puppeteer Vision MCP Server, you'll need Node.js and npm installed on your system. The recommended approach is using npx, which runs the latest version without manual installation.

Environment Setup

Before running the server, set up the required environment variables:

Create a .env file in the directory where you'll run the command, or export the variables in your terminal:

# Required
OPENAI_API_KEY=your_api_key_here

# Optional configurations with defaults
# VISION_MODEL=gpt-4.1
# API_BASE_URL=https://api.openai.com/v1
# TRANSPORT_TYPE=stdio
# PORT=3001
# DISABLE_HEADLESS=true  # Uncomment to see browser activity

Starting the Server

Run the server using npx:

npx -y puppeteer-vision-mcp-server

By default, this starts the server in stdio mode. For HTTP server modes, set TRANSPORT_TYPE=sse or TRANSPORT_TYPE=http in your environment.

Usage

Configuration Options

The server uses these environment variables:

OPENAI_API_KEY: (Required) Your API key for the vision model
VISION_MODEL: (Optional) Vision model to use (default: gpt-4.1)
API_BASE_URL: (Optional) Custom API endpoint URL for alternative providers
TRANSPORT_TYPE: (Optional) Communication protocol - stdio (default), sse, or http
PORT: (Optional) HTTP server port (default: 3001)
DISABLE_HEADLESS: (Optional) Set to true to see the browser in action (default: false)

Communication Modes

The server supports three communication methods:

stdio (Default): Uses standard input/output for direct integration
SSE mode: Server-Sent Events over HTTP at http://localhost:3001/sse
HTTP mode: Streamable HTTP transport with session management at http://localhost:3001/mcp

MCP Tool Integration

To use this as a tool within an MCP-compatible LLM orchestrator, add this configuration:

{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "puppeteer-vision-mcp-server"],
      "env": {
        "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
        // Additional optional environment variables as needed
      }
    }
  }
}

Using the Scraping Tool

The server provides a scrape-webpage tool with these parameters:

url (string, required): The webpage URL to scrape
autoInteract (boolean, optional, default: true): Whether to handle interactive elements
maxInteractionAttempts (number, optional, default: 3): Maximum AI interaction attempts
waitForNetworkIdle (boolean, optional, default: true): Wait for network idle before processing

Response Format

The tool returns results in this structure:

{
  "content": [
    {
      "type": "text",
      "text": "# Page Title\n\nThis is the content..."
    }
  ],
  "metadata": {
    "message": "Scraping successful",
    "success": true,
    "contentSize": 8734
  }
}

How It Works

AI-Driven Interaction

The system uses vision-capable AI models to analyze webpage screenshots and decide on actions like clicking or typing to bypass overlays, cookie banners, captchas, and other interactive elements that might block content.

Content Extraction

After successfully interacting with the page, Mozilla's Readability extracts the main content, which is then sanitized and converted to well-formatted Markdown using Turndown, with special handling for code blocks, tables, and other structured content.

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "web-scraper" '{"command":"npx","args":["-y","puppeteer-vision-mcp-server"],"env":{"OPENAI_API_KEY":"YOUR_OPENAI_API_KEY_HERE"}}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "web-scraper": {
            "command": "npx",
            "args": [
                "-y",
                "puppeteer-vision-mcp-server"
            ],
            "env": {
                "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
            }
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "web-scraper": {
            "command": "npx",
            "args": [
                "-y",
                "puppeteer-vision-mcp-server"
            ],
            "env": {
                "OPENAI_API_KEY": "YOUR_OPENAI_API_KEY_HERE"
            }
        }
    }
}

3. Restart Claude Desktop for the changes to take effect