benchmark-validator

Featured

Benchmark validation procedure for one assigned benchmark-related target. For optimization EXP-* targets, independently classify candidate outcome. For engineering VAL-* or legacy engineering targets, prove or disprove the required benchmark/performance assertion.

AI & Automation 313 stars 46 forks Updated 2 weeks ago Apache-2.0

Install

View on GitHub

Quality Score: 88/100

Stars 20%
83
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark Validator Use this skill for a validation assignment targeting exactly one benchmark-related assertion. For optimization `EXP-*` targets, you are not selecting the winner or optimizing further. You independently decide whether the experiment produced a trustworthy outcome under its contract. For optimization `EXP-*` targets, act as an adversarial tester for the selected candidate. The contract is the minimum bar, not the whole test plan. Before promotion, define and run a compact but comprehensive validation plan that attacks the candidate's likely correctness and performance failure modes. For engineering `VAL-*` or legacy engineering targets, do not use optimization outcome semantics. Pass only when the assigned benchmark, performance, correctness, and guardrail behavior required by the assignment and contract is proven with fresh evidence. ## Inputs Read: - Validation assignment. - The single assigned benchmark-related contract. - `AGENTS.md`. - Experiment ledger path cited by the assignment or contract. - Candidate artifact/ref/patch/checksum cited by the assignment, contract, or ledger. - Measurement protocol, correctness/guardrail commands, protected files, and source/baseline refs cited by the contract. - For optimization targets, also inspect visible runtime resources read-only when useful: changed files, benchmark scripts, fixtures, public outputs, baseline/reference artifacts, verifier-adjacent code, logs, and generated artifacts. Do not m...

Details

Author
Intelligent-Internet
Repository
Intelligent-Internet/zenith
Created
4 months ago
Last Updated
2 weeks ago
Language
Python
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

benchmark-curator

Design, curate, version, and quality-control benchmark or evaluation corpora for Agent Skills and quality workflows, including task taxonomies, discovery/forced/negative controls, adversarial and regression cases, difficulty strata, holdout isolation, provenance, duplication, contamination risk, coverage balance, and immutable benchmark hashes. Use when the user asks to build or maintain an eval dataset, golden set, benchmark suite, regression corpus, holdout, challenge set, or representative test cases. Do not use to execute the model experiment or claim lift (use skill-evaluator), to design the grading rubric (use rubric-designer), to edit the candidate skill (use skill-creator), or to treat leaked/known cases as a clean holdout.

1 Updated today
CometWeb-io
AI & Automation Solid

skill-benchmark

Use when the user runs /skill-benchmark to score agent skills via LLM judges with baseline comparison, regression detection, and trend analysis, or to compare candidate models on a shared task set in a ranked table with per-model spend tracking. Not for release gating — use skill-benchmark-gate.

38 Updated yesterday
OutlineDriven
AI & Automation Listed

agent-benchmark

Use when the user wants a professional, dynamic agent/skill/tool benchmark — compare harnesses, skills, MCPs, CLIs, or workflows on the same tasks with tokens, turns, latency, cost, and success metrics; prove whether a change helps; run ablation-style experiments; or build a reusable bench harness for a repo. Inspired by rigorous same-task evaluation (not GitHub stars).

0 Updated 6 days ago
YosefHayim