Stop Scraping, Start Downloading: The Polite Way to Get Prompt Data

Stop Scraping, Start Downloading: The Polite Way to Get Prompt Data

If you are running a headless browser against /search right now, paging through results one click at a time, parsing HTML that was never meant to be parsed, this post is for you. There is a faster way, and it does not involve pretending to be a human.

We see the traffic. Every week a handful of scrapers hit wikiprompt.org's search page with rotating user agents, random delays, and the occasional retry storm when a selector breaks because we shipped a UI change. It works, kind of, until it does not. Then someone has to go rewrite the scraper, again, because we renamed a CSS class or changed how pagination renders.

Meanwhile the whole catalog, all 55,000+ prompts, is sitting behind a JSON endpoint that answers in milliseconds and does not care how many times you ask.

Why scraping the site is the wrong tool here

Scraping a search UI is a reasonable technique when there is no alternative. It is a bad technique when there is one, and here is why it is specifically bad for a prompt catalog:

None of that is a threat, it is just a description of what scraping costs you. We would rather you skip all of it.

The polite way: the bulk dataset

We publish the dataset precisely so nobody has to scrape us. Hit the manifest first:

curl "https://www.wikiprompt.org/dataset"

It returns total_prompts, the record_fields you can expect on every record, and the pagination scheme. Then pull records from /dataset/prompts:

curl "https://www.wikiprompt.org/dataset/prompts?limit=500"

That single request gets you up to 500 fully-structured prompt records in one response, no headless browser, no HTML to parse, no rendering to wait on. Each record already includes slug, url, title, description, content (the actual prompt text), category, tags, media, model, the structured metadata object, author, original_source, and both timestamps. That is everything you were trying to scrape out of the page, handed to you pre-parsed.

Pagination is keyset, not offset, which is the detail that matters most if you have ever had a scraper silently skip or duplicate records because the underlying list shifted mid-crawl. Follow the next URL in each response until it comes back null. A minimal loop in Python:

import requests

url = "https://www.wikiprompt.org/dataset/prompts?limit=500"

records = []

while url:

resp = requests.get(url).json()

records.extend(resp["records"])

url = resp.get("next")

print(len(records), "prompts pulled, zero pages rendered")

No sleep-and-retry logic, no user-agent rotation, no selector maintenance. The whole catalog, done in a loop that runs in seconds because every response is edge-cached and CORS-enabled (Access-Control-Allow-Origin: *) for direct browser use too.

What you get that scraping never gave you

Beyond speed, the dataset carries signal that was never on the rendered page in a usable form. Take a hand-cut linocut travel poster prompt: the record includes the style array and the aspect ratio right in metadata, the kind of field you'd otherwise have to infer from an image. Or an editorial portrait built around an architectural staircase, where the model field tells you exactly what generated it without you having to guess from the image style. Or a prompt for morphing a character into an arachnid form, which comes with its quality assessment already attached, something no scraper reading the visible page would ever pick up cleanly.

Every record also keeps original_source, the link back to the original tweet or post the prompt came from. That is the piece that makes reuse legitimate: wikiprompt.org aggregates public prompts written by other people, we are not the author of the underlying content, and neither are you if you pull from the dataset. When you use these records, credit both wikiprompt.org as the source and the original_source for the individual prompt. We are not claiming a formal license over other people's posts, we are just the catalog, and we ask you to treat it that way too.

If you need something narrower than the whole dump

The full dataset is for bulk use, training sets, analysis, mirrors. If you actually just need a live query, use the search API instead of scraping /search, it returns the same structured JSON for a single query without you needing to touch the UI at all. If you are building an agent, the MCP server exposes search, retrieval, and even submission as tools. And if you are trying to figure out what is even in scope before writing any code, llms.txt is the map.

The actual ask

Scraping /search gets you a worse copy of data we already hand out for free, slower, more fragile, and heavier on our servers than it needs to be. The dataset gets you the same content, structured, paginated sanely, updated continuously, and it costs you one curl command to start.

If you are currently maintaining a scraper against wikiprompt.org, we are not annoyed, we get it, most sites do not offer this. We do. Point your script at /dataset/prompts instead, delete the scraper, and spend the time you get back on whatever you were actually trying to build.