eval-iterate
SolidIterates on a failing AI/LLM eval (an L2 suite, a golden-set / judge eval, or any eval gating a PR) until it is green AND confirmed, not just luckily passing once. Classifies the failure as a code bug, an eval-definition bug (stale golden item / criteria drift), judge-drift (silent grader version change), or flaky. Applies the minimal fix, then requires N consecutive confirming re-runs — 2 deterministic, 5 for anything a model call grades, since a flat "2" is statistically weak for a stochastic judge. Hard-capped at 5 fix iterations. Refuses to game the eval (Goodhart's Law: skip/delete/overwrite-in-place a case, loosen a threshold) without a second independent check plus `confidence(analysis) >= 90%` and a logged rationale. Composes `ai-engineering`, `confidence`, `critical`, `verify-behavior`. Use when an eval is failing on a PR and needs a real, non-gamed green. Triggers on "this eval is failing", "iterate on this eval", "fix this eval", "get this eval green", "optimize this eval", "/eval-iterate".
Install
Quality Score: 86/100
Skill Content
Details
- Author
- mthines
- Repository
- mthines/agent-skills
- Created
- 4 months ago
- Last Updated
- yesterday
- Language
- JavaScript
- License
- MIT
Similar Skills
Semantically similar based on skill content — not just same category
eval-loop
Use to run a quality check on any generated artifact, or when the user says "evaluate this", "score this draft", "is this good enough", "run the eval", "review this output", "check it against the rubric". Runs generate, judge, revise, re-judge, approve with the judge in a separate context so nothing grades its own work. Logs every cycle including failures. Writes to workspace/evals/log/. Scores your AI system output against a rubric you wrote. Not for writing that rubric (`eval-spec-authoring`), and not the pre-handoff gate on a deliverable this repo produced, which is `artifact-eval`.
skill-iterate
Serial-iterate every scorable skill autonomously overnight via hill-climb. Default invocation drains all .claude/skills/*/evals-bearing skills (minus an embedded skip-list); per-skill budget capped by 1h wall-clock OR 12 iterations, whichever first. Carries autoresearch's NEVER-STOP-mid-loop discipline. Complementary to /skill-evolve (parallel A/B exploration); /skill-iterate is serial exploitation.
evaluating-ai-output
Evaluate non-deterministic LLM/AI output with evals instead of one-shot "it worked" — define expected behavior first, measure pass@k / pass^k, and grade with code / model / human graders. Use when building or changing an AI/LLM feature, an agent, a prompt, a RAG pipeline, or a classifier, where a single good run is not proof of correctness. Complements writing-tests (deterministic logic) and developing-features-tdd.