nasde-benchmark-calibrationlisted

Calibrate assessment rubrics by reviewing agent work in GitHub/GitLab PRs and feeding human comments back into the rubric. Use this skill when the user wants to: - Calibrate, tune, or sanity-check assessment criteria / dimensions of a benchmark - Review trial diffs alongside the LLM-as-a-Judge scores in a PR/MR - Investigate why judge scores feel off, too harsh, too lenient, or misaligned with how a human would grade the code - Pull review comments back from PRs/MRs and turn them into concrete rubric edits Even if the user doesn't say "calibrate" — if they're worried the LLM judge's scores diverge from human judgment, or want to align scores with a real developer's opinion before freezing a benchmark, this skill applies.
NoesisVision/nasde-toolkit · ★ 10 · Code & Development · score 79

Install: claude install-skill NoesisVision/nasde-toolkit

# NASDE Benchmark Calibration Close the loop between the LLM-as-a-Judge and a human reviewer. The judge scores trials against `assessment_criteria.md` (per task) and `assessment_dimensions.json` (benchmark-wide) — but an LLM judge is an imperfect grader, and how it reads the rubric may diverge from how a human grades the code. This skill publishes trial diffs + scores as Pull/Merge Requests for human review, pulls the comments back, and proposes concrete rubric edits. This is the third skill in the benchmark lifecycle: `nasde-benchmark-creator` writes the rubric, `nasde-benchmark-runner` runs trials and scores them, and **this skill calibrates the rubric** against human judgment before the benchmark is frozen. ## Prerequisites - A sink repository that already exists (creation is out of scope). Configure it in `nasde.toml`: ```toml [calibration] repo = "https://github.com/Org/nasde-calibration" # full URL or owner/repo slug # platform = "gitlab" # only needed for a bare slug or a self-hosted host # base_branch = "main" # throttle_sec = 2.0 ``` - The platform CLI for that repo's host: `gh` (GitHub) or `glab` (GitLab), installed and logged in (`gh auth login` / `glab auth login`). The platform is auto-detected from the repo URL host. nasde never handles tokens — the CLI's keyring does. See ADR-010. - `git` on PATH. ## Workflow ### 1. Decide which trials to calibrate Calibration is about the **criteria**, not individual runs. Discuss with the use