agent-evalslisted

# Agent evals An eval answers one question: did the agent do the job correctly? Everything below is a way to ask that question more reliably, more cheaply, and more often. Built from the evals masterclass by Alex Lieberman with Viv (LangChain), 2026: youtube.com/watch?v=zLeG-XJtbIE. The structure (easy, hard, god mode) is theirs. Mistakes in this skill are ours. ## Start by asking the user three things 1. What real job does the agent do? Name the system it touches (CRM, inbox, repo, database). 2. Who in the company is best at that job today? Their judgment is the ground truth. 3. Does the agent run in production yet? If yes, are traces stored anywhere? Pick the mode from the answers. Do not skip to god mode without tasks and verifiers. ## Easy mode: tasks and verifiers A **task** is a small, checkable piece of the real job. A **verifier** says whether that task was done right. 1. Write 10 to 30 tasks that copy what real colleagues do. Examples for a sales agent: - add a note to a CRM record after a call - find a prospect who is not in the obvious table - draft a reply email without sending it 2. For each task, write the verifier. Prefer, in order: - **State check** (a script): the field was empty before and holds the right value after; the tag `email_sent` exists. - **Checklist** (a model or a human): for anything not directly verifiable, break "good" into yes/no questions. "Three paragraphs or fewer?" "Uses the brand tone rules?" 3. Keep every judgment