ai-evals-benchmark-expert
FeaturedExpert guide for LLM & AI Agent automated evaluations (Evals), Promptfoo, DeepEval, Ragas, regression benchmarking, and deterministic assertions / Panduan ahli evaluasi otomatis LLM & AI Agent (Evals), benchmarking regresi, dan pengujian kualitas.
Install
Quality Score: 88/100
Skill Content
Details
- Author
- roedyrustam
- Repository
- roedyrustam/vibes-plug
- Created
- 4 months ago
- Last Updated
- 2 days ago
- Language
- Python
- License
- MIT
Integrates with
Similar Skills
Semantically similar based on skill content — not just same category
ai-prompt-engineering-expert
Expert guide for Prompt Engineering, Chain-of-Thought, few-shot prompting, structured output, prompt injection defense, and automated AI evaluations & regression benchmarking (Promptfoo, DeepEval) / Panduan ahli rekayasa prompt dan evaluasi otomatis AI.
evals-ops
Build and run evals for LLM and agent systems: golden datasets, LLM-as-a-judge with bias control, trajectory/step/outcome scoring, adversarial refuters, and CI regression gates. Triggers on: eval, evals, eval harness, golden dataset, golden set, LLM-as-a-judge, judge rubric, judge bias, judge calibration, Cohen kappa, agreement with human labels, pass@k, pass^k, trajectory eval, step-level eval, tool-call accuracy, regression gate, eval CI, blocking vs advisory eval, faithfulness score, DeepEval, Braintrust, Opik, Langfuse, AgentOps, did my prompt change make it worse, is my agent getting better.
ai-evals
Use when testing an LLM-backed feature, prompt, tool loop or multi-step agent, where the same input can produce different outputs and a prompt or model change can regress behaviour with no code diff. The behaviour spec that precedes the prompt, scenario datasets including adversarial and degradation classes, deterministic assertions over OpenTelemetry traces, calibrated LLM-as-judge, CI gates with baselines, human-in-the-loop, and the production scoring loop that turns incidents into scenarios.