← ClaudeAtlas

llm-output-gatelisted

CI hook that refuses to ship if prompt-eval golden set regresses past threshold or prompt-injection-test fails on HIGH severity
bakw00ds/yakos · ★ 2 · AI & Automation · score 81
Install: claude install-skill bakw00ds/yakos
# LLM Output Gate ## Purpose Pair with `prompt-eval` and `prompt-injection-test` to enforce a CI- side gate: the build fails if the prompt change regresses the golden set beyond `<threshold>` percent on any rubric, OR if any injection payload at or above `<injection-severity>` succeeded. This is the "don't ship a worse prompt" lever. The gate is wired by the `eval-engineer` into the project's CI workflow (GitHub Actions, GitLab pipelines, etc.) and runs on PRs that touch `prompts/**`, `.claude/agents/**`, or any rubric/dataset/corpus path. ## Scope - Wraps `prompt-eval` + `prompt-injection-test` invocations and composes their results into a single CI verdict. - Reads project config from `<project>/eval/.gate.yaml`: - `regression_threshold`: max acceptable per-rubric drop (default 0%). - `aggregate_threshold`: max acceptable aggregate drop (default 2%). - `injection_severity_floor`: severity at which a single hit fails the gate (default `high`). - `paths`: which file changes trigger the gate. - `agents`: which agents to test (defaults to all production agents). - Emits a comment-friendly markdown summary for the PR ("X rubrics regressed; Y injection payloads succeeded; gate: FAIL"). - Exit code: 0 = green, 1 = regressed/jailbroken, 2 = config error. - Designed for `eval-engineer` to wire into CI. The lead does not invoke this skill in normal sessions. ## When to use - As a required CI status check on PRs touching prompt / agent / rubric / dataset p