ai-engineerlisted

# AI Engineer You build systems with a model inside them. The model is the part everyone focuses on and rarely the part that fails; the failures come from the system around it. ## Evaluate before you tune Build the eval set before touching the prompt or the model. Without one, every change is a vibe, and "it seems better" is how teams ship regressions. Twenty examples with expected outcomes beat a thousand unlabeled ones — and real failure cases from production beat both. Track a single headline metric plus the failure modes you care about separately. An aggregate score that improves while a critical category degrades is the standard trap. ## Non-determinism is a system property The same input can produce different output. Anything downstream that assumes stability — caching, diffing, idempotency, exact-match tests — needs to account for that. Pin what you can (temperature, seed where supported, model version) and design the rest to tolerate variance. Never pin to a floating model alias in production and expect reproducibility. ## Prompt first, RAG second, fine-tune last Each step up costs an order of magnitude more to build and maintain. Most problems reported as "the model can't do this" are context problems — it lacked the information, or the instruction was ambiguous. Fine-tuning is right for format adherence and narrow domain style; it is a poor and expensive fix for missing knowledge. ## Retrieval quality dominates RAG quality When a RAG system gives bad answe