ScapeGraph MCP server

ScrapeGraph MCP Server provides seamless integration with the ScapeGraph AI API, enabling language models to leverage advanced AI-powered web scraping capabilities with enterprise-grade reliability. This Model Context Protocol (MCP) server allows your AI assistant to perform various web scraping tasks from simple page extraction to complex multi-page crawling operations.

Available Tools

Core Scraping Tools

• markdownify(website_url: str): Transform any webpage into clean, structured markdown format
• smartscraper(user_prompt: str, website_url: str, number_of_scrolls: int = None, markdown_only: bool = None): Leverage AI to extract structured data from any webpage with support for infinite scrolling
• searchscraper(user_prompt: str, num_results: int = None, number_of_scrolls: int = None): Execute AI-powered web searches with structured, actionable results

Advanced Scraping Tools

• scrape(website_url: str, render_heavy_js: bool = None): Basic scraping endpoint to fetch page content with optional heavy JavaScript rendering
• sitemap(website_url: str): Extract sitemap URLs and structure for any website

Multi-Page Crawling

• smartcrawler_initiate(url: str, prompt: str = None, extraction_mode: str = "ai", depth: int = None, max_pages: int = None, same_domain_only: bool = None): Initiate intelligent multi-page web crawling with two modes:
  • AI Extraction Mode (10 credits per page): Extracts structured data based on your prompt
  • Markdown Conversion Mode (2 credits per page): Converts pages to clean markdown
• smartcrawler_fetch_results(request_id: str): Retrieve results from asynchronous crawling operations

Intelligent Agent-Based Scraping

• agentic_scrapper(url: str, user_prompt: str = None, output_schema: dict = None, steps: list = None, ai_extraction: bool = None, persistent_session: bool = None, timeout_seconds: float = None): Run advanced agentic scraping workflows with customizable steps and structured output schemas

Setup Instructions

To utilize this server, you'll need a ScapeGraph API key. Follow these steps to obtain one:

1. Navigate to the ScapeGraph Dashboard
2. Create an account and generate your API key

Automated Installation via Smithery

For automated installation of the ScrapeGraph API Integration Server using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Claude Desktop Configuration

Update your Claude Desktop configuration file with the following settings (located on the top right of the Cursor page):

(remember to add your API key inside the config)

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

The configuration file is located at:

• Windows: %APPDATA%/Claude/claude_desktop_config.json
• macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json

Cursor Integration

Add the ScrapeGraphAI MCP server in the settings.

Example Use Cases

The server enables sophisticated queries such as:

Single Page Scraping

• "Analyze and extract the main features of the ScapeGraph API"
• "Generate a structured markdown version of the ScapeGraph homepage"
• "Extract and analyze pricing information from the ScapeGraph website with infinite scroll support"
• "Scrape this JavaScript-heavy page with full rendering"

Search and Research

• "Research and summarize recent developments in AI-powered web scraping"
• "Search for the top 5 articles about machine learning frameworks and extract key points"

Multi-Page Crawling

• "Crawl the entire documentation site and convert all pages to markdown"
• "Extract all product information from an e-commerce site up to 3 levels deep"
• "Crawl a blog and extract all article titles, authors, and summaries"

Advanced Agentic Scraping

• "Navigate through a multi-step form and extract the final results"
• "Follow pagination links and compile a complete dataset"
• "Execute a complex workflow with custom extraction schema"

Error Handling

The server implements robust error handling with detailed, actionable error messages for:

• API authentication issues
• Malformed URL structures
• Network connectivity failures
• Rate limiting and quota management

Common Issues

Windows-Specific Connection

When running on Windows systems, you may need to use the following command to connect to the MCP server:

C:\Windows\System32\cmd.exe /c npx -y @smithery/cli@latest run @ScrapeGraphAI/scrapegraph-mcp --config "{\"scrapegraphApiKey\":\"YOUR-SGAI-API-KEY\"}"

This ensures proper execution in the Windows environment.

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "ScrapeGraphAI-scrapegraph-mcp" '{"command":"npx","args":["-y","@smithery/cli@latest","run","@ScrapeGraphAI/scrapegraph-mcp","--config","\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""]}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

3. Restart Claude Desktop for the changes to take effect