← ClaudeAtlas

ragas-evaluationlisted

Use for any question or task about RAG or agent evaluation metrics, including which metrics to use, scoring an answer against its contexts, and telling retrieval problems from generation problems after a change such as chunk size: Ragas faithfulness, response relevancy, context precision and recall, noise sensitivity, factual correctness, tool-call accuracy, agent goal accuracy and rubric scoring.
testland/qa-skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill testland/qa-skills
# ragas-evaluation [rg-gh]: https://github.com/explodinggradients/ragas The model: assemble a dataset (`user_input` + `response` + `retrieved_contexts` + `reference`), import the metrics relevant to the evaluation goal, run `evaluate()`, and inspect per-metric per-row scores (per [rg-gh][rg-gh]). ## When to use - The repo uses LangChain / LlamaIndex / Haystack / direct retriever→LLM RAG pipelines. - You need RAG-specific retrieval + generation quality scoring, or agents-style eval (tool-call and goal accuracy), and want the widest metric variety in OSS LLM-eval. For non-RAG prompt evals, prefer `promptfoo-evaluation`. For pytest-native LLM evals with a managed dashboard, prefer `deepeval-evaluation`. ## Step 1 - Install Per [rg-gh][rg-gh]: ```bash pip install ragas ``` Or from source: ```bash pip install git+https://github.com/explodinggradients/ragas ``` ## Step 2 - Custom metric quickstart Per [rg-gh][rg-gh] (verbatim): ```python import asyncio from openai import AsyncOpenAI from ragas.metrics import DiscreteMetric from ragas.llms import llm_factory # Setup your LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o", client=client) # Create a custom aspect evaluator metric = DiscreteMetric( name="summary_accuracy", allowed_values=["accurate", "inaccurate"], prompt="""Evaluate if the summary is accurate and captures key information. Response: {response} Answer with only 'accurate' or 'inaccurate'.""" ) # Score your application's output async