llm-evaluationlisted

> **QUARANTINE — Community tier.** Patterns here are unvalidated. Read-only tools only. > Verify against official docs (Anthropic, LangSmith) before applying to production. # LLM Evaluation Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. ## When to Use This Skill - Measuring LLM application performance systematically - Comparing different models or prompts - Detecting performance regressions before deployment - Validating improvements from prompt changes - Building confidence in production systems - Establishing baselines and tracking progress over time - Debugging unexpected model behavior ## Core Evaluation Types ### 1. Automated Metrics **Text Generation:** - **BLEU**: N-gram overlap (translation) - **ROUGE**: Recall-oriented (summarization) - **METEOR**: Semantic similarity - **BERTScore**: Embedding-based similarity - **Perplexity**: Language model confidence **Classification:** - Accuracy, Precision/Recall/F1, Confusion Matrix, AUC-ROC **Retrieval (RAG):** - MRR, NDCG, Precision@K, Recall@K ### 2. LLM-as-Judge Use stronger LLMs to evaluate weaker model outputs. **Approaches:** - **Pointwise**: Score individual responses - **Pairwise**: Compare two responses (A/B) - **Reference-based**: Compare to gold standard - **Reference-free**: Judge without ground truth ```python from anthropic import Anthropic from pydantic import BaseModel, Field import json class QualityRating(BaseModel):