Load Testing Best Practices Checklist

# Load Testing taste of a performance-minded reliability engineer - Start every load test from a written goal, such as p95 latency under 300 ms at 500 requests per second, not from a vague "see what breaks". - Write load tests as code with k6, Gatling or Locust and keep them in the same repository as the service. - Model realistic traffic from production logs: the mix of endpoints, payload sizes and think time between requests. - Use an open workload model with a fixed arrival rate when testing services that face the internet, since closed models hide queueing delays. - Run separate test types on purpose: smoke, load at expected peak, stress beyond peak, spike, and a long soak test for memory leaks. - Report latency as percentiles, at least p50, p95 and p99, and never rely on the average alone. - Define pass or fail thresholds in the test script so CI can fail the build automatically. - Run load tests against a production-like environment with the same instance sizes, data volume and configuration. - Seed test data that matches production cardinality, because a tiny database makes every query look fast. - Warm up caches and connection pools before measuring, and exclude the warm-up period from results. - Generate load from machines with enough CPU and network headroom, and watch the load generator's own metrics. - Correlate test results with server-side metrics, traces and logs to find the real bottleneck instead of guessing. - Track error rate and timeouts next to latency; a fast response that returns an error is not a success. - Change one variable per run and record the commit, configuration and environment with each result. - Run a short performance smoke test on every merge to main and the full suite before major releases. - Compare results with a stored baseline and flag regressions larger than an agreed percentage. - Mock or sandbox third-party APIs that charge per call or enforce rate limits, but test your own dependencies for real. - Find the breaking point and document how the system fails: graceful rejection with 429 or 503 is better than collapse. - Verify that autoscaling reacts in time by ramping load gradually and measuring how long new capacity takes to help. - Announce load tests to the team and on-call engineers, and never point them at shared production systems without approval. - Turn findings into capacity plans with clear headroom targets, such as running at no more than 60 percent of the tested limit.