Home / MCP / jcawl4ai MCP Server

jcawl4ai MCP Server

Provides an MCP server to crawl websites via Crawl4ai API and fetch results by task IDs.

other

3stars

View on GitHub

Installation

Add the following to your MCP client configuration file.

Configuration

View docs

{
    "mcpServers": {
        "jcawl4ai-mcp-server-http": {
            "url": "https://mcp.example.com/mcp"
        }
    }
}

You have a Java-based MCP server that integrates with the Crawl4ai API to perform web crawling tasks. It lets you initiate crawls for specific URLs using a chosen strategy, depth, and output format, then retrieve results by task ID. This server is useful when you want to programmatically request crawls and monitor outcomes through a standardized MCP interface.

How to use

To use this MCP server, connect your MCP client to the local or remote server endpoint and call the crawl and task actions exposed by the MCP API. The crawl action lets you start a crawl for a set of URLs with a defined strategy, maximum depth, and output format. The task action retrieves the crawl results for a given task ID. If you already have a crawl task running, you can poll the task endpoint to obtain updates or the final result.

Key capabilities you can leverage include initiating crawls with specific parameters and fetching results by task ID. Ensure your client passes the required parameters in the correct order and handles the JSON results returned by the server.

How to install

Prerequisites: you need Java installed on your machine. You also need Maven to build the project from source if you are starting from a clean checkout.

Step 1. Build the project with Maven.

mvn clean install

Step 2. Run the MCP server jar. The build process outputs a jar file you can execute directly.

java -jar target/jcawl4ai-mcp-server-1.0.0.jar

Step 3. Configure the server to communicate with Crawl4ai. Create or modify the application properties file to include the Crawl4ai base URL and API token.

# File: src/main/resources/application.properties
cawl4ai.base-url=http://your-cral4ai-server-url:11235
cawl4ai.api-token=your-api-token

Additional configuration and notes

You can run the server with a pre-configured MCP server entry to ensure it starts automatically with the needed command. The following configuration shows how to enable the crawl and task actions with an MCP transport via stdio.

{
  "mcpServers": {
    "jcawl4ai-mcp-server": {
      "autoApprove": [
        "crawl",
        "task"
      ],
      "disabled": false,
      "timeout": 60,
      "command": "java",
      "args": [
        "-jar",
        "/path/to/your/jar/file/jcawl4ai-mcp-server-1.0.0.jar"
      ],
      "transportType": "stdio"
    }
  }
}

Configuration and logging details

The server logs its standard output to a log file at the following location: ./target/mcp-stdio-server.log. Review this file for startup messages, errors, and crawl/task activity to troubleshoot issues.

Available tools

crawl

Initiates a crawl for a list of URLs using a specified strategy, maximum depth, and output format; returns a JSON string with crawl details.

task

Retrieves the crawl results for a given task ID; returns a JSON string with the crawl outcome.