benchmark-design

Featured

Design and calibrate a multi-Case capability Benchmark and establish a traceable Formal Baseline.

Web & Frontend 2,099 stars 222 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 93/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark Design Build a multi-Case Benchmark for one Test Agent, calibrate its difficulty with one Run per Case, and record the selected frozen Pilot as the Formal Baseline. This Skill changes the Benchmark, never the Test Agent. It does not run or score the Test Agent. Delegate every evaluation with `run_subagent`, and tell each worker to use `agent-evaluation`. Stop after the Baseline; do not begin optimization. ## Before you start If the request does not identify a Test Agent, target capability, desired baseline score, and Pilot iteration limit, ask for the missing inputs. When they are already supplied, proceed without asking the user to restate them. Treat the current Agent as the **Builder**. A user-specified evaluation `(provider, model_id)` takes priority; otherwise inherit the current Builder Session's complete `Provider` and `Model ID` from the Environment. Never use a Project default as an implicit evaluation runtime. ## Workflow - A **Pilot** is a one-Run-per-Case evaluation used to improve the Benchmark. Unselected Pilot results never enter the Scoreboard; the selected result becomes the Formal Baseline after Freeze. - **Freeze** means the Benchmark revision and evaluation settings stop changing. - A **Formal Baseline** is the accepted result of the selected complete valid Pilot revision, recorded after that exact revision is frozen on one unchanged Agent State version. Follow this order: 1. Validate the Test Agent, target capability, resolved evaluati...

Details

Author
Prism-Shadow
Repository
Prism-Shadow/penguin-harness
Created
1 months ago
Last Updated
today
Language
TypeScript
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category