rl-reward

# RL Reward Construction with OpenJudge Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the `openjudge` library. ## When to Use This Skill - Building scalar rewards for GRPO / REINFORCE rollout scoring - Generating (chosen, rejected) preference pairs for DPO / IPO - Best-of-N candidate selection - Multi-dimensional reward shaping (correctness + safety + format) - Replacing or bootstrapping a reward model with LLM-as-judge ## Step 1 — Choose Your Reward Strategy Use this decision tree **before** writing any code: ``` RL Algorithm + Task type? │ ├── GRPO / REINFORCE — Verifiable task (math, code, structured output) │ └── → POINTWISE ✅ (FunctionGrader, exact score, zero LLM cost) │ ├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization) │ └── → PAIRWISE TOURNAMENT ✅ (compare each rollout vs all others in group, │ reward = net win rate within group) │ ├── DPO / IPO / SLiC — need (chosen, rejected) pairs │ └── → PAIRWISE ✅ (two-way comparison, return winner/loser) │ └── Best-of-N / reranking — rank N candidates └── → LISTWISE ✅ (single call ranks all N at once) ``` ``` Cost constraint? ├── Low budget │ └── FunctionGrader (free) → pointwise; or pairwise with small judge model │ ├── Medium budget │ └── Pointwise: 2–3 LLM graders + WeightedSumAggregator │ └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 c...