agent-harness-evaluation
SolidEvaluate agent harness reliability through replay, restart, cancellation, runtime reconfiguration, concurrent generations, rollback, context pressure, and release gates. Excludes generic tests, surveys, and prompt/model-only benchmarks.
AI & Automation 11 stars
0 forks Updated today MIT
Install
Quality Score: 79/100
Stars 20%
Recency 20%
Frontmatter 20%
Documentation 15%
Issue Health 10%
License 10%
Description 5%
Skill Content
# Agent Harness Evaluation
Bundled commands use `$PLUGIN_ROOT` (`$env:PLUGIN_ROOT` in PowerShell; same path suffix) for the plugin root. Set it once: use the host's plugin-root variable when defined (Claude Code: `PLUGIN_ROOT="$CLAUDE_PLUGIN_ROOT"`), otherwise the absolute path of this skill folder's `../..`.
Evaluate the behavior of the complete agent system, not just the final model answer. Use deterministic post-state and trace evidence wherever possible; use judgment only for residual qualitative questions.
Read `$PLUGIN_ROOT/references/agent-harness-evaluation.md` for suite and reporting guidance and `$PLUGIN_ROOT/references/agent-harness-contracts.md` for artifact, state, and event contracts. Use `$PLUGIN_ROOT/references/agent-harness-landscape.md` only when comparing systems or provider capability surfaces.
## Evidence Boundaries
- Model output is not proof that a side effect was authorized, executed, durable, or correct.
- A transcript is not a checkpoint and cannot prove restart or recovery behavior.
- A diagnostic prompt render does not prove the exact prompt, skill catalog, or
tool schemas seen by another task or runtime origin. Bind the exact rollout
and runtime generation, and capture separately supplied tool schemas
independently.
- Prompt instructions, skills, and scanners do not prove sandbox or policy enforcement.
- Do not infer exactly-once behavior from a clean run. Test duplicate delivery, idempotency, reconciliation, and partial effects.
- Comp...
Details
- Author
- Xopoko
- Repository
- Xopoko/plug-n-skills
- Created
- 2 months ago
- Last Updated
- today
- Language
- Python
- License
- MIT
Similar Skills
Semantically similar based on skill content — not just same category
AI & Automation Solid
agent-harness-engineering
Design agent harnesses for typed loops, runtime reconfiguration, provider/tool/module hot swap, state, policy, cancellation, recovery, and delegation. Excludes prompt-only, generic architecture, vendor CLI/config, and evaluation-only work.
11 Updated today
Xopoko AI & Automation Listed
agent-harness-design
Use when designing tool definitions for a new agent or subagent, an agent shows high retry rates, ambiguous tool invocations, or silent failures, or an existing agent harness needs a quality review.
6 Updated 3 weeks ago
yeaight7