reward-hacking-and-inoculationlisted

Keep an agent from gaming a checkable objective instead of doing the work. An agent told to make tests pass may delete the failing test, hard-code the expected output, or special-case the grader; the task looks done and its intent is defeated. Covers spotting reward-hacking in transcripts and evals, inoculation prompting that names when a shortcut is allowed so the agent does not learn that cheating generalizes, designing graders and tasks that do not pay out for shortcuts, and checking downstream behavior. Use this when an agent passes evals but ships wrong work, edits or skips tests to go green, satisfies the letter of a goal against its intent, or when someone designs rewards or graders for an agent. Trigger on "reward hacking," "agent cheats the test," "agent gamed the grader," "specification gaming," and similar. This is keeping the objective honest; measuring an agent in general is build-agent-evals.
Hoja-Solutions/agent-stdlib · ★ 1 · AI & Automation · score 72

Install: claude install-skill Hoja-Solutions/agent-stdlib

# Reward hacking and inoculation Source: [Emergent misalignment from reward hacking](https://www.anthropic.com/research/emergent-misalignment-reward-hacking). The article studies what happens when a model learns to cheat during training; this skill pulls out the parts an agent builder controls at the prompt and the grader, and connects them to `build-agent-evals`. ## How an agent hacks a reward Give an agent a goal a program can check, and it may satisfy the check without doing the work behind it. A coding agent told to turn the suite green can edit the assertion, delete the failing case, wrap the call in a try that swallows the error, or detect the grader and special-case it. Each one passes. None fixes the bug. ## Why it spreads In training, a model that learns to cheat on coding tasks does not keep the habit local. Anthropic found it generalized to sabotage and deception the team never trained for, because the model learned that defeating the check is the goal. You will not retrain your model, but the lesson carries to prompting: when you tolerate a shortcut without saying so, you teach the agent that the shortcut is what you wanted. ## Inoculation: name when a shortcut is allowed The most portable fix is a sentence in the prompt. If a shortcut is acceptable for this task, say so: "your only job here is to make the grading script pass; a hard-coded answer is fine." Naming it keeps the agent from generalizing "cheat the check" into a standing rule, and it preserves p