Pulse Fetch MCP server for AI agents

This MCP server pulls specific resources from the open internet into context, designed for agent-building frameworks and MCP clients that lack built-in fetch capabilities. It extracts clean, structured content from web pages while minimizing token usage and providing reliable access to protected content through advanced anti-bot capabilities.

Installation

Prerequisites

Node.js (recommended: use the version specified in package.json)
Claude Desktop application (for local setup)
Optional: Firecrawl API key for enhanced scraping capabilities
Optional: BrightData bearer token for web unlocking features

Setting Up with Claude Desktop

Local Setup

You'll need Node.js installed on your machine to run the local version.

Locate your Claude Desktop configuration file:
- macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
- Windows: %APPDATA%\Claude\claude_desktop_config.json
Add this configuration to your Claude Desktop config file:

Minimal configuration (uses native fetch only):

{
  "mcpServers": {
    "pulse-fetch": {
      "command": "npx",
      "args": ["-y", "@pulsemcp/pulse-fetch"]
    }
  }
}

Full configuration (with all optional environment variables):

{
  "mcpServers": {
    "pulse-fetch": {
      "command": "npx",
      "args": ["-y", "@pulsemcp/pulse-fetch"],
      "env": {
        "FIRECRAWL_API_KEY": "your-firecrawl-api-key",
        "BRIGHTDATA_API_KEY": "your-brightdata-bearer-token",
        "STRATEGY_CONFIG_PATH": "/path/to/your/scraping-strategies.md",
        "OPTIMIZE_FOR": "cost",
        "MCP_RESOURCE_STORAGE": "filesystem",
        "MCP_RESOURCE_FILESYSTEM_ROOT": "/path/to/resource/storage"
      }
    }
  }
}

Restart Claude Desktop

Environment Variables

Core Configuration

Environment Variable	Description	Required	Default Value	Example
`FIRECRAWL_API_KEY`	API key for Firecrawl service to bypass anti-bot measures	No	N/A	`fc-abc123...`
`BRIGHTDATA_API_KEY`	Bearer token for BrightData Web Unlocker service	No	N/A	`Bearer bd_abc123...`
`STRATEGY_CONFIG_PATH`	Path to markdown file containing scraping strategy configuration	No	OS temp dir	`/path/to/scraping-strategies.md`
`OPTIMIZE_FOR`	Optimization strategy for scraping: `cost` or `speed`	No	`cost`	`speed`
`MCP_RESOURCE_STORAGE`	Storage backend for saved resources: `memory` or `filesystem`	No	`memory`	`filesystem`
`MCP_RESOURCE_FILESYSTEM_ROOT`	Directory for filesystem storage (only used with `filesystem` type)	No	`/tmp/pulse-fetch/resources`	`/home/user/mcp-resources`
`SKIP_HEALTH_CHECKS`	Skip API authentication health checks at startup	No	`false`	`true`

LLM Configuration for Extract Feature

The extract feature enables intelligent information extraction using LLMs:

Environment Variable	Description	Required	Default Value	Example
`LLM_PROVIDER`	LLM provider: `anthropic`, `openai`, `openai-compatible`	No	N/A	`anthropic`
`LLM_API_KEY`	API key for the chosen LLM provider	No	N/A	`sk-abc123...`
`LLM_API_BASE_URL`	Base URL for OpenAI-compatible providers	No	N/A	`https://api.together.xyz/v1`
`LLM_MODEL`	Specific model to use for extraction	No	See defaults below	`gpt-4-turbo`

Default Models:

Anthropic: claude-sonnet-4-20250514
OpenAI: gpt-4.1-mini
OpenAI-compatible: Provider-specific (must be specified)

Usage

Scrape Tool

The primary tool provided by this MCP server is the scrape tool, which extracts content from web pages.

Parameters:

url (string, required): URL to scrape (e.g., "https://example.com")
timeout (number): Maximum time to wait for page load
maxChars (number): Maximum characters to return (default: 100,000)
startIndex (number): Character index to start output from (for pagination)
saveResult (boolean): Save result as MCP Resource (default: true)
forceRescrape (boolean): Force fresh scrape even if cached (default: false)
cleanScrape (boolean): Clean HTML content by converting to semantic Markdown (default: true)
extract (string): Natural language query for intelligent content extraction (requires LLM configuration)

Result Handling Options

Control how scraped content is handled with the resultHandling parameter:

saveAndReturn (default): Saves content as MCP Resource AND returns it
returnOnly: Returns content without saving - use when you only need content once
saveOnly: Saves content as MCP Resource, returns only resource link - efficient for large content

Usage Tips

The scrape tool handles all web content extraction needs and automatically bypasses anti-bot protection when necessary
Scraped URLs are automatically cached. The tool returns cached content instantly on repeat requests
Use forceRescrape: true to bypass the cache and get fresh content when you know the page has changed
Use maxChars and startIndex parameters to handle large content that exceeds token limits
Configure the timeout parameter (default 60s) for slow-loading sites
Use the extract parameter with natural language queries to extract specific information from pages

Usage Examples

Basic Web Content Extraction

User: "Get the main content from this article: https://example.com/article"
Assistant: I'll extract the content from that article for you.

[Uses scrape tool with onlyMainContent: true]

I've extracted the main article content. The article is titled "Example Article Title" and contains approximately 2,500 words discussing...

Extracting Structured Data

User: "Extract the product information from this e-commerce page: https://shop.example.com/product/123"
Assistant: I'll extract the structured product data from that page.

[Uses scrape tool with format: 'extract' and appropriate schema]

I've extracted the product information:
- Product Name: Example Product
- Price: $99.99
- Rating: 4.5/5 stars
- Description: High-quality example product...

Handling Protected Content

User: "This page is blocking me with CAPTCHA. Can you get the content from https://protected.example.com/data"
Assistant: I'll extract the content from that protected page for you.

[Uses scrape tool with automatic anti-bot bypass]

I successfully bypassed the protection and extracted the content from the page using BrightData's Web Unlocker capabilities.

Intelligent Caching

User: "Get the content from https://example.com/article again"
Assistant: I'll retrieve that content for you.

[Uses scrape tool - automatically returns cached content]

I've retrieved the content from cache (originally scraped 2 hours ago). The article contains...

User: "Actually, I think that article was updated. Can you get the latest version?"
Assistant: I'll fetch a fresh copy of the article for you.

[Uses scrape tool with forceRescrape: true]

I've fetched the latest version of the article. I can see it was indeed updated with new information about...

Scraping Strategy Configuration

The server includes an intelligent strategy system that automatically selects the best scraping method for different websites.

Optimization Modes

The OPTIMIZE_FOR environment variable controls the order and selection of scraping strategies:

COST (default): Optimizes for the lowest cost by trying native fetch first, then Firecrawl, then BrightData
SPEED: Optimizes for faster results by skipping native fetch and starting with more powerful scrapers

Strategy Types

native: Fast native fetch using Node.js fetch API (best for simple pages)
firecrawl: Enhanced content extraction using Firecrawl API (good for complex layouts)
brightdata: Anti-bot bypass using BrightData Web Unlocker (for protected content)

The system automatically learns and applies the best scraping method for specific URL patterns, improving performance over time.

How to install this MCP server

For Claude Code

To add this MCP server to Claude Code, run this command in your terminal:

claude mcp add-json "pulse-fetch" '{"command":"npx","args":["-y","@pulsemcp/pulse-fetch"]}'

See the official Claude Code MCP documentation for more details.

For Cursor

There are two ways to add an MCP server to Cursor. The most common way is to add the server globally in the ~/.cursor/mcp.json file so that it is available in all of your projects.

If you only need the server in a single project, you can add it to the project instead by creating or adding it to the .cursor/mcp.json file.

Adding an MCP server to Cursor globally

To add a global MCP server go to Cursor Settings > Tools & Integrations and click "New MCP Server".

When you click that button the ~/.cursor/mcp.json file will be opened and you can add your server like this:

{
    "mcpServers": {
        "pulse-fetch": {
            "command": "npx",
            "args": [
                "-y",
                "@pulsemcp/pulse-fetch"
            ]
        }
    }
}

Adding an MCP server to a project

To add an MCP server to a project you can create a new .cursor/mcp.json file or add it to the existing one. This will look exactly the same as the global MCP server example above.

How to use the MCP server

Once the server is installed, you might need to head back to Settings > MCP and click the refresh button.

The Cursor agent will then be able to see the available tools the added MCP server has available and will call them when it needs to.

You can also explicitly ask the agent to use the tool by mentioning the tool name and describing what the function does.

For Claude Desktop

To add this MCP server to Claude Desktop:

1. Find your configuration file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Linux: ~/.config/Claude/claude_desktop_config.json

2. Add this to your configuration file:

{
    "mcpServers": {
        "pulse-fetch": {
            "command": "npx",
            "args": [
                "-y",
                "@pulsemcp/pulse-fetch"
            ]
        }
    }
}

3. Restart Claude Desktop for the changes to take effect