llm-testinglisted

# LLM Testing ## This machine already has MLflow tooling for parts of this This environment has the `mlflow:agent-evaluation`, `mlflow:instrumenting-with-mlflow-tracing`, `mlflow:analyzing-mlflow-trace`/`analyzing-mlflow-session`, `mlflow:build-a-scorer`, and `mlflow:querying-mlflow-metrics` skills available — use those directly for trace instrumentation and scored evaluation runs rather than reinventing the harness. This skill covers the testing *methodology*; MLflow's skills cover the *execution mechanics* on this setup. ## What's fundamentally different from classical software testing Non-determinism (same input can produce different output) and no single "correct" output for open-ended generation — you're testing distributions and properties, not exact-match assertions, most of the time. ## Evaluation approaches - **Golden dataset with reference answers**: for tasks with a knowable correct answer (extraction, classification via LLM, structured output), compare against references using exact match, semantic similarity (embedding cosine similarity), or a rubric. - **LLM-as-judge**: use a second LLM call to score the output against a rubric (relevance, correctness, tone) when no single reference answer exists — but validate the judge itself against human-labeled examples periodically, since a miscalibrated judge silently produces meaningless scores. See `mlflow:build-a-scorer`. - **Human evaluation**: still the ground truth for subjective quality — sample a subset for hum