agent-harness-evaluation

Solid

Evaluate agent harness reliability through replay, restart, cancellation, runtime reconfiguration, concurrent generations, rollback, context pressure, and release gates. Excludes generic tests, surveys, and prompt/model-only benchmarks.

AI & Automation 11 stars 0 forks Updated today MIT

Install

View on GitHub

Quality Score: 79/100

Stars 20%
36
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Agent Harness Evaluation Bundled commands use `$PLUGIN_ROOT` (`$env:PLUGIN_ROOT` in PowerShell; same path suffix) for the plugin root. Set it once: use the host's plugin-root variable when defined (Claude Code: `PLUGIN_ROOT="$CLAUDE_PLUGIN_ROOT"`), otherwise the absolute path of this skill folder's `../..`. Evaluate the behavior of the complete agent system, not just the final model answer. Use deterministic post-state and trace evidence wherever possible; use judgment only for residual qualitative questions. Read `$PLUGIN_ROOT/references/agent-harness-evaluation.md` for suite and reporting guidance and `$PLUGIN_ROOT/references/agent-harness-contracts.md` for artifact, state, and event contracts. Use `$PLUGIN_ROOT/references/agent-harness-landscape.md` only when comparing systems or provider capability surfaces. ## Evidence Boundaries - Model output is not proof that a side effect was authorized, executed, durable, or correct. - A transcript is not a checkpoint and cannot prove restart or recovery behavior. - A diagnostic prompt render does not prove the exact prompt, skill catalog, or tool schemas seen by another task or runtime origin. Bind the exact rollout and runtime generation, and capture separately supplied tool schemas independently. - Prompt instructions, skills, and scanners do not prove sandbox or policy enforcement. - Do not infer exactly-once behavior from a clean run. Test duplicate delivery, idempotency, reconciliation, and partial effects. - Comp...

Details

Author
Xopoko
Repository
Xopoko/plug-n-skills
Created
2 months ago
Last Updated
today
Language
Python
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category