ai_gymlisted

Train AI agent prompts by MEASUREMENT instead of vibes — stand up an eval harness (real prompt + real LLM + STUBBED tools = zero side effects), author scenarios with pass/fail rubrics, judge transcripts with subagents, aggregate a scorecard by dimension, run a TOURNAMENT of prompt variants, kill LLM noise with N-runs, and ship only the change measurement proved. Use whenever the user wants to improve/refine/test the system prompt of an AI agent, chatbot, or LLM pipeline; compare two prompt versions; find why an agent hallucinates, leaks data, or escalates wrong; build a regression eval for a prompt; or run "agent battles / a prompt tournament". Triggers on requests like "make my agent's prompt better", "this bot keeps making things up", "which prompt version is better", "write a test for my bot's behavior", "I need an eval for my agents" — even without the word "eval". Target stack: TS/Node + vitest; the LLM provider is pluggable (your project supplies the client). Works with any subagent-capable coding agent
luisroquette/ai-gym · ★ 0 · AI & Automation · score 67

Install: claude install-skill luisroquette/ai-gym

# AI Gym — Eval-Driven Prompt Refinement (Agent Prompt Tournament) An agent's prompt is code: it deserves measurement, not gut feeling. This is the complete method to **prove** a prompt change actually improves behavior — not just "feels better" — before it reaches production. ## Why this exists (read before touching code) LLMs are not deterministic, even at `temperature: 0`. "I tweaked the prompt and it looks good" is a sample of one, dominated by noise. A plausible edit can be a **net wash** — the win on one case eaten by a regression on another you never saw. The only defense is: an objective per-behavior rubric + an independent judge + **N-runs comparing baseline-vs-edited per case**. Without it, you ship a regression disguised as an improvement. The harness also isolates the agent from the world: **real prompt + real LLM + STUBBED tools**. The tools record the call and return the scenario's canonical context — they never create a real ticket, order, or record. You measure the agent's reasoning without touching anything real. ## The cycle in 7 steps (the map) `harness` → `scenarios` → `judge` → `scorecard` → `attack weakness` → `N-runs` → `ship` 0. **Harness** — real prompt + real LLM + stubbed tools (zero side effects). 1. **Scenarios** — one behavior per scenario, a `must` / `must_not` rubric. 2. **Baseline** — run them all, produce transcripts. 3. **Judge** — Claude subagents score 0-3 against the rubric. 4. **Scorecard** — aggregate by dimension, rank the weakn