Resilient Scraper

Anti-block web scraping with automatic fallback chain. Never lose data to a 403 again.

Quick Start

# Scrape a URL with full fallback chain
python3 scripts/resilient-scraper.py fetch "https://example.com/page"

# Scrape with specific output format
python3 scripts/resilient-scraper.py fetch "https://example.com" --format markdown

# Check domain health (success rate, block history)
python3 scripts/resilient-scraper.py status "example.com"

# View scrape history and success rates
python3 scripts/resilient-scraper.py stats

# Clear cache for a domain
python3 scripts/resilient-scraper.py cache clear "example.com"

# Batch scrape from URL list
python3 scripts/resilient-scraper.py batch urls.txt --delay 5

# Test all fallback levels against a URL
python3 scripts/resilient-scraper.py diagnose "https://example.com/page"

Fallback Chain

The scraper tries each level in order, escalating on failure:

Level 1: Simple fetch (requests + rotated User-Agent)
    ↓ on 403/429/timeout
Level 2: Enhanced fetch (cookies, referer spoofing, session reuse)
    ↓ on failure
Level 3: Browser fetch (headless Chromium via Playwright/Selenium)
    ↓ on failure
Level 4: Cached response (return last successful scrape if < 24h old)
    ↓ on cache miss
Level 5: Error report with diagnosis

User-Agent Rotation

50+ real browser User-Agent strings, rotated per request. Includes:

• Chrome (Windows, Mac, Linux) — latest 5 versions
• Firefox (Windows, Mac, Linux) — latest 5 versions
• Safari (Mac, iOS)
• Edge (Windows)

Rate Limiting

Built-in per-domain rate limiting:

• Default: 1 request per 3 seconds per domain
• Configurable via --delay flag
• Respects Retry-After headers
• Exponential backoff on repeated failures (5s → 10s → 20s → 60s)

Cache

Successful responses cached at ~/.openclaw/workspace/scraper-cache/:

• Cache key: URL hash + format
• Default TTL: 24 hours
• Max cache size: 500MB (LRU eviction)

Domain Health Tracking

Per-domain success/failure stats at ~/.openclaw/workspace/scraper-stats.jsonl:

• Success rate percentage
• Average response time
• Block frequency
• Last successful scrape timestamp
• Recommended scrape level (auto-learned)