Browser Scraping & Search MCP server

This MCP-powered browser automation agent enables you to search Google, navigate to webpages, and intelligently scrape content from various websites, all controlled through an AI interface. The system combines Playwright for browser control with Mistral AI for processing the extracted information.

Requirements

• Python 3.8+
• Playwright
• MCP (Model Controlled Program)
• Mistral AI API key

Installation

1. Clone the repository:

git clone https://github.com/yourusername/browser-automation-agent.git
cd browser-automation-agent

2. Install dependencies:

pip install -r requirements.txt

3. Install Playwright browsers:

playwright install

4. Create a .env file in the project root and add your Mistral AI API key:

MISTRAL_API_KEY=your_api_key_here

Usage

Running the Server and Client

You need to run both the server and client components in separate terminals:

# Terminal 1 - Run the server
python main.py

# Terminal 2 - Run the client
python client.py

How It Works

Once both components are running:

1. Enter your query when prompted
2. The agent will:
   • Search Google for relevant results
   • Navigate to the top result
   • Scrape content based on the website type
   • Save screenshots and content to files
   • Return processed information

Available Scraping Capabilities

The agent has specialized scraping strategies for different website types:

• GitHub repositories: Extracts README content and code blocks
• Stack Overflow: Extracts questions, answers, comments, and code blocks
• Documentation pages: Optimized for documentation content and code examples
• Generic websites: Extracts paragraph text and code blocks

Output Files

The agent generates two types of output files with timestamps:

• final_page_YYYYMMDD_HHMMSS.png: Screenshot of the final page state
• scraped_content_YYYYMMDD_HHMMSS.txt: Extracted text content from the page

Troubleshooting

• Connection Issues: Ensure both server and client are running in separate terminals
• Playwright Errors: Make sure browsers are installed with playwright install
• API Key Errors: Verify your Mistral API key is correctly set in the .env file
• Path Errors: Update the path to main.py in client.py if needed

How to add this MCP server to Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > MCP and click "Add new global MCP server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "cursor-rules-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "cursor-rules-mcp"
            ]
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explictly ask the agent to use the tool by mentioning the tool name and describing what the function does.