home / skills / openclaw / skills / bright-data

bright-data skill

safe

This skill enables web scraping and Google search using Bright Data API to retrieve structured results and page content as markdown.

npx playbooks add skill openclaw/skills --skill bright-data

Review the files below or copy the command above to add this skill to your agents.

Files (4)

SKILL.md

1.3 KB

---
name: brightdata
description: Web scraping and search via Bright Data API. Requires BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE. Use for scraping any webpage as markdown (bypassing bot detection/CAPTCHA) or searching Google with structured results.
---

# Bright Data - Web Scraping & Search

Direct API access to Bright Data's Web Unlocker and SERP APIs.

## Setup

**1. Get your API Key:**
Get a key from [Bright Data Dashboard](https://brightdata.com/cp).

**2. Create a Web Unlocker zone:**
Create a zone at brightdata.com/cp by clicking "Add" (top-right), selecting "Unlocker zone".

**3. Set environment variables:**
```bash
export BRIGHTDATA_API_KEY="your-api-key"
export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name"
```

## Usage

### Google Search
Search Google and get structured JSON results (title, link, description).
```bash
bash scripts/search.sh "query" [cursor]
```
- `cursor`: Optional page number for pagination (0-indexed, default: 0)

### Web Scraping
Scrape any webpage as markdown. Bypasses bot detection and CAPTCHA.
```bash
bash scripts/scrape.sh "url"
```

## Output Formats

### Search Results
Returns JSON with structured `organic` array:
```json
{
  "organic": [
    {"link": "...", "title": "...", "description": "..."}
  ]
}
```

### Scrape Results
Returns clean markdown content from the webpage.

Overview

This skill integrates with Bright Data to perform reliable web scraping and Google search via Bright Data's Unlocker and SERP APIs. It requires BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE to bypass bot detection and CAPTCHA and to return clean, structured outputs. Use it to get search result JSON or to convert arbitrary webpages into readable markdown.

How this skill works

The skill calls Bright Data's Web Unlocker to fetch page HTML while evading anti-bot systems and CAPTCHAs, then converts the cleaned content into Markdown. For search, it queries Bright Data's SERP endpoint and returns structured JSON with organic results (title, link, description). Both features are exposed as simple scripts or API-style commands that accept a query, cursor (page), or URL.

When to use it

You need to scrape pages protected by bot detection or CAPTCHAs.
You want Google search results in a structured JSON format for processing.
You need readable, markdown-formatted content from arbitrary webpages.
You want pagination-aware SERP scraping using a cursor parameter.
You must integrate scraping or search into pipelines without managing proxies.

Best practices

Set BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE as environment variables before running.
Respect target site terms of service and robots.txt where applicable.
Limit request rates and use pagination (cursor) for large result sets.
Validate and sanitize scraped content before downstream processing.
Monitor API usage and errors to avoid unexpected costs or blocks.

Example use cases

Fetch first-page Google results for a keyword and ingest titles, links, and snippets into a data pipeline.
Scrape a dynamic, JavaScript-heavy product page and convert it into markdown for documentation or analysis.
Archive article content from sites that usually block direct scraping for research or backup.
Automate competitive monitoring by scheduling repeated SERP queries with cursors for deep pages.
Extract structured links and descriptions to build a curated link list or feed.

FAQ

What environment variables are required?

You must set BRIGHTDATA_API_KEY and BRIGHTDATA_UNLOCKER_ZONE before using the skill.

What output formats are available?

Search returns structured JSON with an organic array (title, link, description). Scraping returns clean markdown content of the page.

Can this bypass CAPTCHAs and bot detection?

Yes. The Web Unlocker is designed to bypass common bot detection and CAPTCHA systems, but success depends on target site complexity and Bright Data's current capabilities.