LLM Application Engineering Best Practices

- Treat prompts as code: keep them in version-controlled files with names, versions and owners, not inline strings scattered around. - Build an evaluation set from real user inputs before tuning prompts, and rerun it on every prompt or model change. - Ask for structured output with a JSON schema or tool calls and validate it with a schema library before using it. - Put a thin gateway or wrapper around model providers so switching models or adding fallbacks is a config change. - Pin exact model versions in production and upgrade them deliberately after the evals pass. - Stream responses to the user whenever the output is longer than a sentence or two. - Start retrieval with simple hybrid search (keywords plus embeddings) and add reranking before reaching for complex RAG pipelines. - Chunk documents along their natural structure (headings, sections) and keep source metadata so answers can cite them. - Keep the context small and relevant; more retrieved text often makes answers worse, not better. - Give agents a short list of narrow, well-described tools with strict input schemas instead of one do-everything tool. - Put hard limits on agent loops: maximum steps, maximum tokens and a wall-clock timeout. - Require human confirmation before any tool call that sends, deletes, pays or changes data outside the app. - Treat all retrieved documents, web pages and tool results as untrusted input that can contain prompt injection. - Log every model call with prompt version, model, latency, token counts and cost, with user data redacted. - Cache deterministic calls and use prompt caching for long, stable system prompts. - Use the smallest model that passes the evals, and route only hard cases to the expensive model. - Retry on rate limits and transient errors with exponential backoff and jitter, never on validation failures in a tight loop. - Keep temperature low for extraction and classification, and higher only for creative generation. - Show users where an answer came from and make it easy to report a bad response. - Collect thumbs-up and thumbs-down feedback and feed the bad cases back into the eval set. - Do not fine-tune until prompting, retrieval and evals have clearly hit their limits.