evaluate
FeaturedEvaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.
Install
Quality Score: 93/100
Skill Content
Details
- Author
- softspark
- Repository
- softspark/ai-toolkit
- Created
- 4 months ago
- Last Updated
- today
- Language
- Python
- License
- MIT
Integrates with
Similar Skills
Semantically similar based on skill content — not just same category
monitor-rag-quality
Use this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully. Trigger on "my RAG gives wrong answers", "is my retrieval any good", "the chatbot makes things up", "evaluate my RAG", "improve RAG accuracy". Diagnose whether the failure is in retrieval or generation - they need different fixes.
add-llm-evals
Use this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Covers offline (CI) and online (production LLM-as-a-judge) evaluation.
rag-eval-guardrails
Build a verified eval harness for a RAG/LLM feature plus PII/PHI-leakage guardrails, gated by checks that actually run. Scores a precomputed predictions file (so it runs with ZERO API access) on groundedness, citation validity, retrieval hit@k, answer F1/exact-match, refusal rate, and latency; compares to config thresholds and a baseline to catch regressions; and fails the build on PII/PHI leakage. Use when the user wants to evaluate or regression-test an AI/RAG feature, measure hallucination/groundedness, add an eval gate to CI, or scan prompts/answers/logs for leaked identifiers. Triggers: "RAG evaluation", "LLM eval", "eval harness", "hallucination", "groundedness", "PII/PHI leakage", "guardrails", "regression testing for AI features".