set-up-ab-testinglisted

# A/B test LLM changes in production Offline evals tell you a change *should* be better; a production experiment tells you it *is*, on real users and real traffic. For anything user-facing, run a controlled rollout instead of flipping the switch for everyone. ## When to use this vs offline evals - **Offline evals** (`add-llm-evals`): fast, cheap, run in CI, catch regressions before deploy. Do these first, always. - **A/B / online experiment:** use when the change is user-facing and the offline metric does not fully capture success (helpfulness, engagement, task completion, downstream conversion). Offline says "passes"; A/B says "users actually do better." ## Set it up 1. **Pick one variable.** New prompt OR new model OR new retrieval, not all at once, or you cannot attribute the result. 2. **Define the metric up front.** The primary success metric (task completion, thumbs-up rate, downstream action) plus guardrail metrics (cost, latency, error rate) that must not regress. Decide the win condition before you start. 3. **Split traffic** deterministically by user/session (a feature-flag / experiment tool: LaunchDarkly, GrowthBook, or your platform's built-in). Keep a user on one variant for consistency. 4. **Instrument both arms** with the same tracing + online evals + feedback capture (`instrument-llm-observability`, `collect-user-feedback`), tagged with the variant, so every metric is comparable across arms. 5. **Run until it is significant**, not until it looks good. Sma