evallisted

# Eval — AI/LLM Evaluation ## Activate When - User invokes `/godmode:eval` - User says "evaluate my model", "benchmark this" - User says "LLM-as-judge", "compare models", "run evals" - When deploying or updating any AI/LLM component ## Workflow ### Step 1: Evaluation Discovery ```bash # Find existing eval infrastructure find . -name "eval*" -o -name "benchmark*" \ -o -name "judge*" | grep -v node_modules # Check for eval frameworks grep -l "deepeval\|ragas\|promptfoo\|braintrust" \ package.json pyproject.toml requirements.txt \ 2>/dev/null ``` ``` EVALUATION DISCOVERY: System: <which AI system to evaluate> Type: LLM prompt | RAG pipeline | AI agent | model Trigger: new system | model change | prompt change Dimensions: correctness, relevance, faithfulness, safety, format compliance, latency, cost IF no baseline exists: establish baseline first IF model changed: run full regression suite IF prompt changed: run targeted eval on affected dims ``` ### Step 2: Evaluation Dataset Design ``` DATASET SOURCES: | Source | Count | Quality | |-----------------|-------|-----------| | Golden set | <N> | Highest | | Production logs | <N> | Realistic | | Synthetic | <N> | Scalable | | Adversarial | <N> | Edge cases| THRESHOLDS: Minimum golden set: 50 examples Minimum per category: 10 examples Adversarial coverage: >= 20% of total set IF dataset < 50: results not statistically reliable IF any category < 10: expand before trustin