skill-eval

Featured

Measure whether a skill helps a named task or needs revision or removal. Use when: a bounded routing or coding evaluation is requested; conformance alone cannot show benefit.

AI & Automation 446 stars 41 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 91/100

Stars 20%
88
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# /skill-eval Answer one named maintenance decision: **retain, revise, remove, or insufficient evidence**. Choose the measurement that can answer that decision, use the caller's accepted cases and resource envelope, make one scoped recommendation, and stop. A completed evaluation does not require a positive difference. This is an optional specialist. The repository's selected runner owns execution and bounds; native results own measurements; BD and Git retain their authority. Do not add a core skill, AO evaluation command, scheduler, dashboard, second tracker, or mandatory review merely to run an experiment. ## Choose the question | Caller decision | Measurement | What it can establish | |---|---|---| | Does loading this skill change a specific observable act? | Behavioral probe with `scripts/probe-skill.sh` | Behavior change on that scenario; not correct code or productivity | | Does this package or version improve engineering outcomes at acceptable cost? | Repository-selected controlled coding comparison, such as `evals/skills-rpi` | Endpoint outcomes and cost on selected tasks; independent completion only when required exact-subject evidence exists | | Does a qualified memory update help later work? | Separate frozen-versus-updated memory transfer test | Narrow later-task reuse evidence with skill and runtime held fixed | | What happened in ordinary runs? | Existing native accounting and acceptance evidence | Observational failures, repairs and cost; not causal skill b...

Details

Author
boshu2
Repository
boshu2/agentops
Created
10 months ago
Last Updated
today
Language
Go
License
Apache-2.0

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

skill-evaluator

Design and evaluate Agent Skill experiments that measure whether a skill improves model behavior, discovery, task success, reliability, cost, or latency relative to a no-skill/prior-version baseline, including host/model comparisons, judge agreement, quality-cost Pareto trade-offs, and runtime drift under frozen measurement identity. Use when the user asks to benchmark, A/B test, evaluate, compare, prove, regress-test, or measure a skill across supported harnesses, or to prepare executable real-host suites when execution is unavailable. Do not use for static package/routing audits (use skill-auditor), to create/edit a skill (use skill-creator), to fabricate real-host results, or to claim universal superiority from one configuration.

1 Updated today
CometWeb-io
AI & Automation Listed

skill-eval

Use when you want to test, measure, or compare skill effectiveness. Supports eval, compare, metrics, baseline capture, and regression detection.

7 Updated 3 days ago
joris887
Code & Development Listed

skill-eval-harness

Write, evaluate and maintain agent skills with measured evidence rather than assertion. Covers SKILL.md structure and frontmatter, writing a description that triggers reliably without firing on neighbouring topics, progressive disclosure through references and scripts, and building eval suites across three families - usefulness with skill on versus off, discovery, and adversarial discovery. Use when creating a new skill, reviewing or debugging one that does not trigger or does not help, deciding whether a skill earns its context cost, writing an eval suite or grader, or interpreting skill eval results. Also use when asked whether a skill is any good.

0 Updated 4 days ago
Premchand006