benchmark-methodology
SolidRules for measuring or quoting any benchmark number in this repo. Load before running the bench tier, writing a delta into a PR or CHANGELOG, or touching anything under benchmarks/.
Install
Quality Score: 81/100
Skill Content
Details
- Author
- jgravelle
- Repository
- jgravelle/jcodemunch-mcp
- Created
- 7 months ago
- Last Updated
- today
- Language
- Python
- License
- NOASSERTION
Integrates with
Similar Skills
Semantically similar based on skill content — not just same category
benchmark-analysis
Takes one of the user's own metrics and checks it against a stated benchmark source, returning a clear over/under read and what that gap actually means. Use when the user wants to know if a number (churn rate, CAC, conversion rate, NPS) is good or bad relative to a real reference point, not just the number in isolation. Boundary: this skill does not have a live connection to any benchmark database. It compares against whatever source the user supplies, or discloses plainly when it is using general public knowledge instead.
benchmark
Progress-ceiling verdict gate (part of `/spec execute`'s improve tail) — decide whether ONE landed change was actually BENEFICIAL (moved or held the capability-benchmark ceiling without breaking the regression floor, and is worth its complexity), then emit a single BENEFICIAL/NOT-BENEFICIAL verdict. Composes (never forks) the existing instruments: /eval (the regression floor — probes stay green) + the capability-benchmark ceiling delta (.oh/evals/capability/RESULTS.md vs. the counterfactual). Machinery added with no benchmark movement is NOT-BENEFICIAL by definition. Distinct from /audit implementation (per-unit promotability = floor) — this is the ceiling: did the harness get BETTER, not just not-broken. TRIGGER when: a change has landed and the loop needs a benefit-vs-counterfactual verdict before the cycle repeats; the improve tail of `/spec execute` runs; "was <change> beneficial", "score the capability benchmark", "benchmark this cycle".
benchmark-implement
Run an approved PRD through the existing receipt-backed implement workflow, analyze its runtime session with a fresh independent evaluator, emit a deterministic process report, and compare it with an earlier run. Use when the user invokes "$benchmark-implement", asks to benchmark the implement skill or harness, rerun a fixed PRD after harness changes, analyze implementation-session efficiency, or compare baseline and candidate harness behavior.