A responsible, production-grade scraping engine
A Python + Playwright web scraper kit with exponential backoff, configurable rate limiting, rotating user agents and CSV export. Built with respect for websites in mind.
What's included
- Scraper engine — generic with retry + exponential backoff + jitter
- Rate limiting — configurable delays between requests
- Rotating user agents — reduced blocking
- CSV export — automatic headers, append mode
- Config-driven —
config.yaml for all settings
- Demo scraper — working example against a public books site
- Legal disclaimer — responsible scraping practices documented
Quick Start
pip install -r requirements.txt
playwright install
python scraper.py
Stack
Python 3.9+ · Playwright · PyYAML · fake-useragent
Use cases
- Price monitoring
- Data collection for research
- Lead generation (public data)
- Automated testing of public sites
