add-runner-eval

Featured

Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.

AI & Automation 114,742 stars 19046 forks Updated today MIT

Install

View on GitHub

Quality Score: 92/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Add a Runner Eval Use this skill for the **Runner Evals** family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md). Locate the Paperclip checkout using `PAPERCLIP_ROOT` when supplied, or `git rev-parse --show-toplevel` from a checkout. From outside Git, inspect the workspace roots (for example `~/paperclipai/paperclip`) and verify that the selected root contains `packages/paperclip-runner` and `tests/runner-e2e`. Locate `paperclip-evals` using `PAPERCLIP_EVALS_ROOT` or a discovered sibling; a worktree's parent directory need not contain that repository. Read `doc/evals.md` and `packages/paperclip-runner/docs/runner-protocol-live-evals.md`, then inspect the nearest existing case, roster, schema, and report test before editing. Definitions and authored cases belong in the sibling `paperclip-evals/evals/paperclip-runner`; Runner integration, aggregation, viewer, and publication behavior belongs in `packages/paperclip-runner`. Keep the control-plane boundary explicit in names and documentation. The sibling eval README is the concrete map: cases live under `cases/`, company fixtures under `fixtures/`, runtime/model settings under `configs/`, selections under `rosters/`, and maintained campaign membership under `campaigns/live-direct-full.json`. Update inventory/coverage mappings when the program requires them; a new file alone does not join t...

Details

Author
paperclipai
Repository
paperclipai/paperclip
Created
8 months ago
Last Updated
today
Language
TypeScript
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category