benchmark-design

Featured

Design and calibrate a multi-Case capability Benchmark with repeated independent evaluations and a traceable baseline.

AI & Automation 199 stars 23 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 93/100

Stars 20%
77
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark Design Create and calibrate a multi-Case Benchmark that discovers a specified Test Agent's capability boundary. Own the public Statements, private Rubrics, Case set, configuration, and final baseline. Do not modify the Test Agent State, launch the Test Agent directly, or score a Case yourself. ## Before you start Require a Test Agent and the capability to measure. If either is missing, ask the user. A Benchmark run also requires a top-level Session with `run_subagent` and the current Agent must have `agent-evaluation` installed. If the Skill is missing or this Session is already a subagent, stop and ask the user to install the Skill or start a top-level Session. Do not begin a partial Benchmark. ## Boundaries Access only the explicit Test Agent and Benchmark paths. Do not inspect another Agent, Project configuration files, Agent Evaluator State, Evaluator Workspace, or Evaluator Trace. Consume only each Evaluator's terminal protocol response and the returned Test Session id. Use the Environment's App Data Dir and the explicit Test Agent id: ```text PROJECT_DIR = <app_data_dir> PROJECT_ID = <basename_of_project_dir> PENGUIN_HOME = <parent_of_project_dir> TEST_AGENT_DIR = <app_data_dir>/agents/<test_agent_id> BENCHMARK_DIR = <app_data_dir>/agents/<test_agent_id>/benchmarks/<benchmark_id> SCOREBOARD = <benchmark_dir>/scoreboard.yaml ``` Derive a semantic Benchmark id when the user does not supply one. Require `agent_state/system_config.yaml`; its top-level `v...

Details

Author
Prism-Shadow
Repository
Prism-Shadow/penguin-harness
Created
1 weeks ago
Last Updated
today
Language
TypeScript
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category