ScapeGraph MCP server for AI agents

ScrapeGraph MCP Server is a production-ready implementation of the Model Context Protocol that connects language models to ScrapeGraph AI's web scraping API. It enables AI assistants to extract data from websites, convert webpages to markdown, perform multi-page crawling, and execute complex scraping workflows with minimal configuration.

Installation

Get Your API Key

First, you'll need to obtain an API key from the ScrapeGraph Dashboard.

Install with Smithery (Recommended)

The easiest way to install the server is using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Claude Desktop Configuration

After installing with Smithery, update your Claude Desktop configuration:

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

The configuration file is located at:

Windows: %APPDATA%/Claude/claude_desktop_config.json
macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json

Manual Installation

For manual installation:

Clone the repository:

git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
cd scrapegraph-mcp

Install the package:

pip install -e .

Set your API key:

# macOS/Linux
export SGAI_API_KEY=your-api-key-here

# Windows (PowerShell)
$env:SGAI_API_KEY="your-api-key-here"

Available Tools

Core Scraping Tools

`markdownify`

Transform any webpage into clean markdown:

markdownify(website_url: str)

`smartscraper`

Extract structured data with AI:

smartscraper(
    user_prompt: str,
    website_url: str,
    number_of_scrolls: int = None,
    markdown_only: bool = None
)

`searchscraper`

Execute AI-powered web searches:

searchscraper(
    user_prompt: str,
    num_results: int = None,
    number_of_scrolls: int = None
)

Advanced Scraping Tools

`scrape`

Basic page content fetching:

scrape(website_url: str, render_heavy_js: bool = None)

`sitemap`

Extract website structure:

sitemap(website_url: str)

Multi-Page Crawling

`smartcrawler_initiate`

Start multi-page crawling:

smartcrawler_initiate(
    url: str,
    prompt: str = None,
    extraction_mode: str = "ai",
    depth: int = None,
    max_pages: int = None,
    same_domain_only: bool = None
)

`smartcrawler_fetch_results`

Retrieve crawling results:

smartcrawler_fetch_results(request_id: str)

`agentic_scrapper`

Run complex scraping workflows:

agentic_scrapper(
    url: str,
    user_prompt: str = None,
    output_schema: dict = None,
    steps: list = None,
    ai_extraction: bool = None,
    persistent_session: bool = None,
    timeout_seconds: float = None
)

Usage Examples

Single Page Scraping

"Convert the ScrapeGraph documentation page to markdown"
"Extract all product names, prices, and ratings from this e-commerce page"
"Scrape this infinite scroll page with 5 scrolls and extract all items"

Search and Research

"Research and summarize recent developments in AI-powered web scraping"
"Search for the top 5 articles about machine learning frameworks and extract key insights"

Website Analysis

"Extract the complete sitemap structure from the ScrapeGraph website"
"Discover all URLs on this blog site"

Multi-Page Crawling

"Crawl the entire documentation site and extract all API endpoints with descriptions"
"Convert all pages in the blog to markdown up to 2 levels deep"

Google ADK Integration

You can integrate the ScrapeGraph MCP server with Google's Agent Development Kit:

import os
from google.adk.agents import LlmAgent
from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams
from mcp import StdioServerParameters

# Path to the server script
SERVER_SCRIPT_PATH = "/path/to/scrapegraph-mcp/src/scrapegraph_mcp/server.py"

root_agent = LlmAgent(
    model='gemini-2.0-flash',
    name='scrapegraph_assistant_agent',
    instruction='Help the user with web scraping using ScrapeGraph AI.',
    tools=[
        MCPToolset(
            connection_params=StdioConnectionParams(
                server_params=StdioServerParameters(
                    command='python3',
                    args=[SERVER_SCRIPT_PATH],
                    env={'SGAI_API_KEY': os.getenv('SGAI_API_KEY')},
                ),
                timeout=300.0,)
            )
        )
    ],
)

Troubleshooting

Common Issues

"ScrapeGraph client not initialized"

Make sure your API key is set correctly

"Error 401: Unauthorized"

Verify your API key is valid

"Error 402: Payment Required"

Add more credits to your ScrapeGraph account

SmartCrawler not returning results

Continue polling smartcrawler_fetch_results() until status is "completed"

Windows-Specific Connection

On Windows systems, you may need to use:

C:\Windows\System32\cmd.exe /c npx -y @smithery/cli@latest run @ScrapeGraphAI/scrapegraph-mcp --config "{\"scrapegraphApiKey\":\"YOUR-SGAI-API-KEY\"}"

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "ScrapeGraphAI-scrapegraph-mcp" '{"command":"npx","args":["-y","@smithery/cli@latest","run","@ScrapeGraphAI/scrapegraph-mcp","--config","\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""]}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

3. Restart Claude Desktop for the changes to take effect