Polite Web Harvester Prompt for Claude Code
From Wikiprompt, the free prompt encyclopedia
Oct 5, 2026
Polite Web Harvester Prompt for Claude Code A prompt to generate a respectful web harvester that avoids bans by following robots.txt, rate limiting, and deduplication, with a 7-step pipeline.
Prompt ContentSave
🌐
Build a polite web harvester in Python: seed, fetch, render, clean, extract, dedupe, export. Respect robots.txt, rate-limit every host, honor Retry-After on 429. Use Playwright only for JS pages. Validate every record and drop duplicates by content hash. Output: dataset.jsonl. Show me the plan before writing code.
Sign in to see the full prompt
Continue with:
By logging in, you agree to our Terms of Use and Privacy Policy
Usage
This prompt is designed for use with coding. Copy the prompt content above and paste it into your preferred AI tool.
For best results, you may customize the placeholders (indicated by square brackets or capital letters) with your specific requirements.
References
- Category: coding Prompts
- Source: https://x.com/Shelpid_WI3M/status/2107123530571678047
Talk
0 comments