ai-engineeringlisted
Install: claude install-skill RaveenDi/Skilloadout
# AI Engineering
AI engineering is the gap between "the model did the right thing in my test prompt" and "this
feature works for real users, at scale, reliably, and we'd know within minutes if it broke."
It borrows from software engineering (testing, versioning, observability, rollout) and applies it
to a component that is probabilistic and prompt-sensitive instead of deterministic.
## 1. Evals before features
- **Write the eval set before you tune the prompt.** A representative set of real (or
realistic) inputs with a way to score the output — exact match, rubric-graded, LLM-as-judge, or
a human review queue — is the only way to know if a change helped or hurt.
- **Capture failures from production into the eval set.** Every bug report is a missing test
case; the eval set should grow every time something breaks.
- **Score on the actual task outcome**, not surface similarity to a reference answer. If the task
is "extract these 5 fields," score field-by-field accuracy, not BLEU-style text overlap.
- **Use an LLM judge carefully**: give it a rubric, few-shot examples of each score, and
spot-check its judgments against humans regularly — judges drift and can be gamed by verbose
but wrong answers.
## 2. Choosing how knowledge gets to the model
Pick the cheapest technique that clears the eval bar — don't reach for fine-tuning by default:
| Need | Reach for |
|---|---|
| Teach a response *format* or *style* | Few-shot examples in the prompt |
| Give access to infor