add-llm-evalslisted

# Add evaluations to an LLM app Observability tells you *what happened*; evaluation tells you *whether it was good*. Add both an offline suite (runs in CI on a fixed dataset) and, optionally, online scoring (grades production traffic). ## Pick the eval type per metric Two families, use both where relevant: - **Reference-based** (you have a ground-truth answer): exact/fuzzy match, semantic similarity, ROUGE/BLEU - cheap, deterministic. Good for classification, extraction, closed QA. - **Reference-free / LLM-as-a-judge** (open-ended output): a strong model scores the output against a rubric (faithfulness, relevance, coherence, safety). Scalable proxy for human judgment - but calibrate it (see caveats). For RAG specifically, use the standard quartet: **faithfulness, answer relevance, context precision, context recall** (Ragas implements these). ## Build the offline suite (do this first) 1. **Curate a dataset** - 20-100 representative inputs with expected outputs or rubrics. Pull real cases from production traces (this is where observability + eval connect). Version it in the repo. 2. **Choose a framework** - `promptfoo` (YAML, great for prompt/RAG + CI), `DeepEval` (pytest-style, 40+ metrics), or `Ragas` (RAG metrics). Pick one; don't hand-roll. 3. **Define pass criteria** - per-metric thresholds (e.g. faithfulness ≥ 0.8). Fail the build if a threshold regresses. 4. **Wire into CI** - run on every PR that touches prompts/models/retrieval. Store scores so you can see trend