advanced-evaluationlisted

# Advanced Evaluation **Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops. ## When to Activate - Building LLM-as-judge systems for LLM outputs - Comparing multiple model responses to select the best one - Establishing consistent quality standards across evaluation teams - Debugging evaluation systems that show inconsistent results - Designing A/B tests for prompt or model changes - Creating rubrics specifically for LLM or human/LLM hybrid judges - Analyzing correlation between automated and human judgments Do not activate this skill for adjacent work owned by other skills: - General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`. - Tool API contracts for evaluation tools: `tool-design`. ## Core Concepts ### The Evaluation Taxonomy Select between two primary approaches based on whether ground truth exists: **Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation. **Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better on