agent-evaluation

Featured

Run one specified Test Agent on one specified Benchmark Case exactly once, privately score that execution, and return one protocol result.

AI & Automation 2,099 stars 222 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 93/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Agent Evaluation Handle one evaluation request from a `run_subagent` caller: run the specified Test Agent on one Benchmark Case once, score that execution privately, and return one protocol result. The top-level Benchmark Designer or Optimizer owns all Case and Run loops, concurrency, and follow-up handling. This worker handles no other Case or Run, launches no evaluator or subagent, modifies no Agent or Benchmark, and never writes `scoreboard.yaml`. Use the Penguin CLI only to launch the specified Test Agent; do not use it to create another phase, designer, optimizer, or evaluator. Operate silently. Call tools without progress messages. Across all streamed and final responses, the only worker-authored text must be the final plain protocol YAML. Emit no narration, headings, Markdown fences, summaries, private scoring details, or other text. ## Before you start Use this Skill only for a complete request from a `run_subagent` caller. If the request is incomplete or inconsistent, return `invalid_request` through the protocol instead of asking the user a question. ## Contract Require exactly one value for every field below: ```text protocol_version: 1 case_id: <case_id> run: <1_based_run_index> expected_version: <tested_agent_state_version> test_agent_id: <test_agent_id> benchmark_id: <benchmark_id> provider: <provider> model_id: <model_id> ``` One request represents one Test Agent execution. The `run` value identifies that execution; it is not a repeat count. `provide...

Details

Author
Prism-Shadow
Repository
Prism-Shadow/penguin-harness
Created
1 months ago
Last Updated
today
Language
TypeScript
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category