ragas-evaluationlisted
Install: claude install-skill testland/qa-skills
# ragas-evaluation
[rg-gh]: https://github.com/explodinggradients/ragas
The model: assemble a dataset (`user_input` + `response` +
`retrieved_contexts` + `reference`), import the metrics relevant to the
evaluation goal, run `evaluate()`, and inspect per-metric per-row
scores (per [rg-gh][rg-gh]).
## When to use
- The repo uses LangChain / LlamaIndex / Haystack / direct
retriever→LLM RAG pipelines.
- You need RAG-specific retrieval + generation quality scoring, or
agents-style eval (tool-call and goal accuracy), and want the widest
metric variety in OSS LLM-eval.
For non-RAG prompt evals, prefer `promptfoo-evaluation`.
For pytest-native LLM evals with a managed dashboard, prefer
`deepeval-evaluation`.
## Step 1 - Install
Per [rg-gh][rg-gh]:
```bash
pip install ragas
```
Or from source:
```bash
pip install git+https://github.com/explodinggradients/ragas
```
## Step 2 - Custom metric quickstart
Per [rg-gh][rg-gh] (verbatim):
```python
import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory
# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)
# Create a custom aspect evaluator
metric = DiscreteMetric(
name="summary_accuracy",
allowed_values=["accurate", "inaccurate"],
prompt="""Evaluate if the summary is accurate and captures
key information.
Response: {response}
Answer with only 'accurate' or 'inaccurate'."""
)
# Score your application's output
async