home / mcp / mcp web scraper server

MCP Web Scraper Server

Provides web search, content extraction, and metadata tools via MCP endpoints for automated scraping and discovery.

Installation

Add the following to your MCP client configuration file.

Configuration

{
  "mcpServers": {
    "aniruddha1202-mcp-web-scraper": {
      "url": "https://mcp-web-scraper-production.up.railway.app",
      "headers": {
        "HOST": "0.0.0.0",
        "PORT": "8000",
        "LOG_LEVEL": "INFO"
      }
    }
  }
}

You can deploy a production-ready MCP Server for web scraping and search, then connect clients to run advanced web searches, extract content, and gather structured data from web pages. This MCP Server provides web search, smart search, news search, and content extraction tools, all accessible via HTTP or local stdio interfaces.

How to use

Your MCP client will call the available tools to perform actions like web search, smart search, or content extraction. Use the HTTP endpoint to connect remotely, or run the server locally as a stdio service and connect via the provided command. Typical usage involves selecting a tool, passing a query (and optional parameters like mode or result limits), and then handling the structured results returned by the server.

Available tools

web_search

Search the web for a query and return results.

smart_search

Perform intelligent search in quick, standard, or comprehensive modes.

search_and_scrape

Search and automatically extract full content from results.

news_search

Search specifically for news articles with dates and sources.

scrape_html

Scrape HTML content with optional CSS selectors.

extract_links

Extract all links with optional filtering.

extract_metadata

Retrieve page metadata and Open Graph tags.

scrape_table

Extract table data from webpages.

extract_article

Clean article extraction by removing ads and navigation.