benchmark-methodology

Solid

Rules for measuring or quoting any benchmark number in this repo. Load before running the bench tier, writing a delta into a PR or CHANGELOG, or touching anything under benchmarks/.

AI & Automation 2,680 stars 366 forks Updated today NOASSERTION

Install

View on GitHub

Quality Score: 81/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
76
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark methodology Authority: `docs/harness/ARCHAEOLOGY.md` (the benchmark rows), `benchmarks/METHODOLOGY.md`, `benchmarks/REPRODUCING.md`, CLAUDE.md Practice 4, `docs/harness/FINDINGS.md` F-10, F-13, F-17. Rules, each a pointer, none restated: 1. **Never hand-type a number.** Every figure comes from a run in THIS session or the CI-captured artifact `benchmarks/jcm_reference.json` (Practice 4). The comparators read that file; there is no estimator. 2. **Per row, never per total** (F-13): the total hid one cause behind another. Diff threshold ids one by one. 3. **Five mirrors move together** (Practice 4): `results.md`, `METHODOLOGY.md`, README, `provenance/measured.json`, `REPRODUCING.md`; `tests/test_provenance.py` is the gate and `--reference` rewrites them. 4. **The reference is captured where the gate runs** (F-13): `benchmark.yml` dispatch with `reference=true`; a dev-box number differs by CRLF, walk order and a HOME ledger (F-17). 5. **Deterministic configuration** is the bench tier itself: `--offline`, pinned corpora (`benchmarks/tasks.json`), the no-network fixture. Add no flags. 6. **A refusal is not a zero**: an absent value prints `n/a`. 7. **Floors live only in `harness/thresholds.json`**; read one with `uv run python -m harness threshold <id>`. 8. `benchmarks/schema_baseline.json` is the only source for schema-token figures (CLAUDE.md "Tier-switch pricing"); the harness counts a different payload and the two never agree d...

Details

Author
jgravelle
Repository
jgravelle/jcodemunch-mcp
Created
7 months ago
Last Updated
today
Language
Python
License
NOASSERTION

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

benchmark-analysis

Takes one of the user's own metrics and checks it against a stated benchmark source, returning a clear over/under read and what that gap actually means. Use when the user wants to know if a number (churn rate, CAC, conversion rate, NPS) is good or bad relative to a real reference point, not just the number in isolation. Boundary: this skill does not have a live connection to any benchmark database. It compares against whatever source the user supplies, or discloses plainly when it is using general public knowledge instead.

1 Updated today
sidchaudhary
AI & Automation Solid

benchmark

Progress-ceiling verdict gate (part of `/spec execute`'s improve tail) — decide whether ONE landed change was actually BENEFICIAL (moved or held the capability-benchmark ceiling without breaking the regression floor, and is worth its complexity), then emit a single BENEFICIAL/NOT-BENEFICIAL verdict. Composes (never forks) the existing instruments: /eval (the regression floor — probes stay green) + the capability-benchmark ceiling delta (.oh/evals/capability/RESULTS.md vs. the counterfactual). Machinery added with no benchmark movement is NOT-BENEFICIAL by definition. Distinct from /audit implementation (per-unit promotability = floor) — this is the ceiling: did the harness get BETTER, not just not-broken. TRIGGER when: a change has landed and the loop needs a benefit-vs-counterfactual verdict before the cycle repeats; the improve tail of `/spec execute` runs; "was <change> beneficial", "score the capability benchmark", "benchmark this cycle".

38 Updated today
mifunedev
AI & Automation Listed

benchmark-implement

Run an approved PRD through the existing receipt-backed implement workflow, analyze its runtime session with a fresh independent evaluator, emit a deterministic process report, and compare it with an earlier run. Use when the user invokes "$benchmark-implement", asks to benchmark the implement skill or harness, rerun a fixed PRD after harness changes, analyze implementation-session efficiency, or compare baseline and candidate harness behavior.

1 Updated today
yansfil