agent-evals

# Agent Evals Create repeatable checks so agent behavior improves safely over time. ## When to Use This Skill Use this skill when: - Shipping new agent features or changing prompts - Adding CI gates for agent quality and safety - Building regression suites for tool-calling agents - Measuring LLM output quality at scale - Validating RAG retrieval accuracy ## Prerequisites - Python 3.10+ - An LLM API key (OpenAI, Anthropic, etc.) - pytest or a custom eval harness - Optional: Braintrust, Promptfoo, or LangSmith account ## Evaluation Layers ### Unit Evals — Prompt-Level Correctness Test individual prompt → response quality: ```python # evals/test_unit.py import json import pytest from agent import generate_response CASES = json.load(open("evals/fixtures/unit_cases.json")) @pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"]) def test_prompt_correctness(case): result = generate_response(case["prompt"], model=case.get("model", "default")) # Exact match for structured output if case.get("expected_json"): assert json.loads(result) == case["expected_json"] # Substring match for free-text for keyword in case.get("must_contain", []): assert keyword.lower() in result.lower(), f"Missing: {keyword}" for keyword in case.get("must_not_contain", []): assert keyword.lower() not in result.lower(), f"Unexpected: {keyword}" ``` Golden dataset format: ```json [ { "id": "calc-01", "prompt": "What is 15% tip on $42.50?",...