benchmark-validator

Solid

Benchmark validation procedure for one assigned benchmark-related target. For optimization EXP-* targets, independently classify candidate outcome. For engineering VAL-* or legacy engineering targets, prove or disprove the required benchmark/performance assertion.

AI & Automation 280 stars 40 forks Updated 2 weeks ago Apache-2.0

Install

View on GitHub

Quality Score: 86/100

Stars 20%
82
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark Validator Use this skill for a validation assignment targeting exactly one benchmark-related assertion. For optimization `EXP-*` targets, you are not selecting the winner or optimizing further. You independently decide whether the experiment produced a trustworthy outcome under its contract. For optimization `EXP-*` targets, act as an adversarial tester for the selected candidate. The contract is the minimum bar, not the whole test plan. Before promotion, define and run a compact but comprehensive validation plan that attacks the candidate's likely correctness and performance failure modes. For engineering `VAL-*` or legacy engineering targets, do not use optimization outcome semantics. Pass only when the assigned benchmark, performance, correctness, and guardrail behavior required by the assignment and contract is proven with fresh evidence. ## Inputs Read: - Validation assignment. - The single assigned benchmark-related contract. - `AGENTS.md`. - Experiment ledger path cited by the assignment or contract. - Candidate artifact/ref/patch/checksum cited by the assignment, contract, or ledger. - Measurement protocol, correctness/guardrail commands, protected files, and source/baseline refs cited by the contract. - For optimization targets, also inspect visible runtime resources read-only when useful: changed files, benchmark scripts, fixtures, public outputs, baseline/reference artifacts, verifier-adjacent code, logs, and generated artifacts. Do not m...

Details

Author
Intelligent-Internet
Repository
Intelligent-Internet/zenith
Created
3 months ago
Last Updated
2 weeks ago
Language
Python
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category