Python Web Scraper Kit (Playwright + Retry/Backoff)

A responsible, production-grade scraping engine

A Python + Playwright web scraper kit with exponential backoff, configurable rate limiting, rotating user agents and CSV export. Built with respect for websites in mind.

What's included

• Scraper engine — generic with retry + exponential backoff + jitter
• Rate limiting — configurable delays between requests
• Rotating user agents — reduced blocking
• CSV export — automatic headers, append mode
• Config-driven — config.yaml for all settings
• Demo scraper — working example against a public books site
• Legal disclaimer — responsible scraping practices documented

Quick Start

pip install -r requirements.txt
playwright install
python scraper.py

Stack

Python 3.9+ · Playwright · PyYAML · fake-useragent

Use cases

• Price monitoring
• Data collection for research
• Lead generation (public data)
• Automated testing of public sites