WebScraping.AI MCP Server provides a Model Context Protocol implementation that integrates with WebScraping.AI service for advanced web data extraction capabilities. It enables LLM assistants to extract structured data, answer questions about web content, and retrieve HTML with JavaScript rendering support.
Run the server directly using npx:
env WEBSCRAPING_AI_API_KEY=your_api_key npx -y webscraping-ai-mcp
# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server
# Install dependencies
npm install
# Run
npm start
Note: Requires Cursor version 0.45.6+
The WebScraping.AI MCP server can be configured in two ways in Cursor:
Project-specific Configuration (recommended for team projects): Create a .cursor/mcp.json file in your project directory:
{
"servers": {
"webscraping-ai": {
"type": "command",
"command": "npx -y webscraping-ai-mcp",
"env": {
"WEBSCRAPING_AI_API_KEY": "your-api-key",
"WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
}
}
}
}
Global Configuration (for personal use across all projects): Create a ~/.cursor/mcp.json file in your home directory with the same configuration format as above.
For Windows users experiencing issues, try using:
cmd /c "set WEBSCRAPING_AI_API_KEY=your-api-key && npx -y webscraping-ai-mcp"
Add this to your claude_desktop_config.json:
{
"mcpServers": {
"mcp-server-webscraping-ai": {
"command": "npx",
"args": ["-y", "webscraping-ai-mcp"],
"env": {
"WEBSCRAPING_AI_API_KEY": "YOUR_API_KEY_HERE",
"WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5"
}
}
}
}
WEBSCRAPING_AI_API_KEY: Your WebScraping.AI API key
WEBSCRAPING_AI_CONCURRENCY_LIMIT: Maximum number of concurrent requests (default: 5)WEBSCRAPING_AI_DEFAULT_PROXY_TYPE: Type of proxy to use (default: residential)WEBSCRAPING_AI_DEFAULT_JS_RENDERING: Enable/disable JavaScript rendering (default: true)WEBSCRAPING_AI_DEFAULT_TIMEOUT: Maximum web page retrieval time in ms (default: 15000, max: 30000)WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT: Maximum JavaScript rendering time in ms (default: 2000)# Required
export WEBSCRAPING_AI_API_KEY=your-api-key
# Optional - customize behavior (default values)
export WEBSCRAPING_AI_CONCURRENCY_LIMIT=5
export WEBSCRAPING_AI_DEFAULT_PROXY_TYPE=residential # datacenter or residential
export WEBSCRAPING_AI_DEFAULT_JS_RENDERING=true
export WEBSCRAPING_AI_DEFAULT_TIMEOUT=15000
export WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT=2000
Ask questions about web page content.
{
"name": "webscraping_ai_question",
"arguments": {
"url": "https://example.com",
"question": "What is the main topic of this page?",
"timeout": 30000,
"js": true,
"js_timeout": 2000,
"wait_for": ".content-loaded",
"proxy": "datacenter",
"country": "us"
}
}
Extract structured data from web pages based on instructions.
{
"name": "webscraping_ai_fields",
"arguments": {
"url": "https://example.com/product",
"fields": {
"title": "Extract the product title",
"price": "Extract the product price",
"description": "Extract the product description"
},
"js": true,
"timeout": 30000
}
}
Get the full HTML of a web page with JavaScript rendering.
{
"name": "webscraping_ai_html",
"arguments": {
"url": "https://example.com",
"js": true,
"timeout": 30000,
"wait_for": "#content-loaded"
}
}
Extract the visible text content from a web page.
{
"name": "webscraping_ai_text",
"arguments": {
"url": "https://example.com",
"js": true,
"timeout": 30000
}
}
Extract content from a specific element using a CSS selector.
{
"name": "webscraping_ai_selected",
"arguments": {
"url": "https://example.com",
"css_selector": ".product-price",
"js": true,
"timeout": 30000
}
}
Extract content from multiple elements using CSS selectors.
{
"name": "webscraping_ai_selected_multiple",
"arguments": {
"url": "https://example.com",
"css_selectors": {
"title": "h1",
"price": ".price",
"description": ".description"
},
"js": true,
"timeout": 30000
}
}
Get information about your WebScraping.AI account.
{
"name": "webscraping_ai_account",
"arguments": {}
}
The following options can be used with all scraping tools:
timeout: Maximum web page retrieval time in ms (15000 by default, maximum is 30000)js: Execute on-page JavaScript using a headless browser (true by default)js_timeout: Maximum JavaScript rendering time in ms (2000 by default)wait_for: CSS selector to wait for before returning the page contentproxy: Type of proxy, datacenter or residential (residential by default)country: Country of the proxy to use (US by default). Supported countries: us, gb, de, it, fr, ca, es, ru, jp, kr, incustom_proxy: Your own proxy URL in "http://user:password@host:port" formatdevice: Type of device emulation. Supported values: desktop, mobile, tableterror_on_404: Return error on 404 HTTP status on the target page (false by default)error_on_redirect: Return error on redirect on the target page (false by default)js_script: Custom JavaScript code to execute on the target pageThere are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.
If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.
To add a global MCP server go to Cursor Settings > MCP and click "Add new global MCP server".
When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:
{
"mcpServers": {
"cursor-rules-mcp": {
"command": "npx",
"args": [
"-y",
"cursor-rules-mcp"
]
}
}
}
To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.
Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.
The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.
You can also explictly ask the agent to use the tool by mentioning the tool name and describing what the function does.