Four iron rules for building real evals

1. Don't build your question bank from today's model weaknesses: model capabilities are uneven. If you specifically pick questions the model gets wrong today, you're measuring the fingerprint flaws of this specific version, not task difficulty. Next-gen model ships and your bank is useless. 2. Even the strongest model at max thinking must score far below 100: if your test set lets the strongest model easily get full marks, the questions are too easy and you can't tell whether later changes improved or degraded things. 3. Prefer programs as judges, then LLMs: anything that can be checked with code (unit tests, JSON Schema validation) must not use an LLM judge. When an LLM judge is required, don't use vague 1-5 scores; write fact-checkable checklists or do blind A/B comparison. 4. Must distinguish real improvement from random noise: if scores vary every run, the questions are ambiguous or the environment left residue files letting the AI cheat.