annotate-traces-for-reviewlisted

# Annotate and review traces Automated metrics are downstream of one thing: a human deciding what "good" means. For specialized or high-stakes domains (finance, health, legal), and for early-stage apps, structured human review of real traces is the single highest-ROI activity. It produces the golden labels every other eval depends on, and it surfaces failure modes you did not know to look for. ## Set up the review loop 1. **Pull a sample of traces** to review. Stratify (by topic, difficulty, low online-eval score, thumbs-down) so reviewers see the interesting cases, not 100 easy ones. 2. **Give reviewers the full context** the model had: input, retrieved docs, tool results, output. Redact PII first for regulated data (`redact-pii-for-tracing`). 3. **Use a simple, consistent schema:** pass/fail (or a small rubric score) + a **free-text failure reason** + a category tag. The free-text is where you discover new failure modes; the categories let you count them. 4. **Use the tooling** rather than spreadsheets where possible: annotation queues exist in Langfuse, Phoenix, Opik, LangSmith and let annotations attach to the trace. ## Do error analysis (not just labeling) The point is not a score, it is understanding. After a review pass: - **Read the free-text reasons and cluster them** into failure categories (retrieval miss, hallucination, formatting, refusal, tone, ...). Count each. - **Fix the biggest category first.** A few categories usually explain most failures. - This is