openjudge

# OpenJudge Skill Build evaluation pipelines for LLM applications using the `openjudge` library. ## When to Use This Skill - User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.) - User wants to compare two or more models and rank them - User wants to design a scoring rubric and automate evaluation - User wants to analyze evaluation results statistically - User wants to build a reward model or quality filter ## Sub-documents — Read When Relevant | Topic | File | Read when… | |-------|------|------------| | Grader selection & configuration | `graders.md` | User needs to pick or configure an evaluator | | Batch evaluation pipeline | `pipeline.md` | User needs to run evaluation over a dataset | | Auto-generate graders from data | `generator.md` | No rubric yet; generate from labeled examples | | Analyze & compare results | `analyzer.md` | User wants win rates, statistics, or metrics | Read the relevant sub-document **before** writing any code. ## Install ```bash pip install py-openjudge ``` ## Architecture Overview ``` Dataset (List[dict]) │ ▼ GradingRunner ← orchestrates everything │ ├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank ├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank └─► Grader C ... │ ├─► Aggregator (optional) ← combine multiple grader scores into one │ └─► RunnerResult ← {grader_nam...