playbooks

home / skills / beshkenadze / claude-skills-marketplace / wget-reader

wget-reader skill

/skills/utility/wget-reader

This skill fetches web content or downloads files using wget, validating URLs, honoring headers, timeouts, and retries to deliver data or files.

npx playbooks add skill beshkenadze/claude-skills-marketplace --skill wget-reader

Review the files below or copy the command above to add this skill to your agents.

Files (1)
SKILL.md
2.1 KB
---
name: wget-reader
description: Fetch data from URLs. Use when asked to download content, fetch remote files, or read web data.
version: 1.0.0
---

# Wget URL Reader

## Overview

Fetches content from URLs using wget command-line tool. Supports downloading files, reading web pages, and retrieving API responses.

## Instructions

1. When user provides a URL to read or fetch:
   - Validate the URL format
   - Use wget with appropriate flags based on content type

2. For reading content to stdout (display):
   ```bash
   wget -qO- "<URL>"
   ```

3. For downloading files:
   ```bash
   wget -O "<filename>" "<URL>"
   ```

4. For JSON API responses:
   ```bash
   wget -qO- --header="Accept: application/json" "<URL>"
   ```

5. Common wget flags:
   - `-q`: Quiet mode (no progress output)
   - `-O-`: Output to stdout
   - `-O <file>`: Output to specific file
   - `--header`: Add custom HTTP header
   - `--timeout=<seconds>`: Set timeout
   - `--tries=<n>`: Number of retries
   - `--user-agent=<agent>`: Set user agent

## Examples

### Example: Read webpage content
**Input:** "Read the content from https://example.com"
**Command:**
```bash
wget -qO- "https://example.com"
```

### Example: Download a file
**Input:** "Download the file from https://example.com/data.json"
**Command:**
```bash
wget -O "data.json" "https://example.com/data.json"
```

### Example: Fetch API with headers
**Input:** "Fetch JSON from https://api.example.com/data"
**Command:**
```bash
wget -qO- --header="Accept: application/json" "https://api.example.com/data"
```

### Example: Download with timeout and retries
**Input:** "Download with 30 second timeout"
**Command:**
```bash
wget --timeout=30 --tries=3 -O "output.txt" "<URL>"
```

## Guidelines

### Do
- Always quote URLs to handle special characters
- Use `-q` flag to suppress progress bars in scripts
- Add `--timeout` for unreliable endpoints
- Respect robots.txt and rate limits

### Don't
- Use `--no-check-certificate` unless necessary
- Fetch URLs without validating format first
- Ignore HTTP error codes in responses
- Store credentials in command history

Overview

This skill fetches content from URLs using the wget command-line tool. It supports reading web pages to stdout, downloading files to disk, and retrieving API responses with custom headers. Use it to reliably pull remote content in scripts or interactive sessions.

How this skill works

The skill validates URL format, then constructs a wget command with flags tailored to the requested action. For quick reads it uses quiet output to stdout, for file downloads it specifies an output filename, and for API calls it adds headers like Accept: application/json. It also exposes common options for timeouts, retries, and user agent customization.

When to use it

  • Read a web page or API response directly to your terminal or pipeline
  • Download a remote file and save it with a specific filename
  • Fetch JSON or other API data while sending custom HTTP headers
  • Add timeouts and retry behavior for unstable endpoints
  • Integrate remote fetches into shell scripts or automation pipelines

Best practices

  • Always quote URLs to handle spaces and special characters
  • Use -q and -O- for clean stdout output in scripts
  • Set --timeout and --tries for unreliable networks
  • Respect robots.txt, rate limits, and remote usage policies
  • Avoid --no-check-certificate unless you understand the security risk
  • Validate URL format before attempting to fetch

Example use cases

  • Read the HTML of https://example.com: wget -qO- "https://example.com"
  • Save a file to disk: wget -O "data.json" "https://example.com/data.json"
  • Retrieve JSON API output: wget -qO- --header="Accept: application/json" "https://api.example.com/data"
  • Download with retry and timeout: wget --timeout=30 --tries=3 -O "output.txt" "<URL>"
  • Pipe fetched content into other tools: wget -qO- "<URL>" | jq '.'

FAQ

How do I fetch JSON from an API?

Use wget -qO- --header="Accept: application/json" "<URL>" and pipe or save the output as needed.

What flags help in unreliable networks?

Use --timeout=<seconds> and --tries=<n> to control how long and how many times wget will retry.