ai-agents-empirical-probe-toolkit

Solid

Prove-it methods for this repo. Six recipes for runtime-contract probes, guard and threshold calibration, eval A/B, docs-vs-reality audits, reproduce-on-main CI triage, and negative-control test design, each with a worked example from repo history. Use when you say `probe the runtime contract`, `calibrate this guard`, `prove it empirically`. Do NOT use for the portability battle plan (use `ai-agents-portability-campaign`) or evidence standards (use `ai-agents-validation-and-qa`).

AI & Automation 38 stars 13 forks Updated today MIT

Install

View on GitHub

Quality Score: 77/100

Stars 20%
53
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# AI Agents Empirical Probe Toolkit <!-- vendor-portability: contributor-facing knowledge pack for the rjmurillo/ai-agents repo itself; intentionally references upstream paths (.agents/, .claude/, scripts/, build/) because its audience is repo contributors, not plugin consumers (issue #2050) --> This skill is the method library for replacing assumption with measurement. Every recipe exists because an assumption shipped here and became an incident. The house rule, from the #2290 payload-format incident: the cost of a probe is 15 minutes; the cost of assumption is a P0 (`.serena/memories/copilot-hooks-observations.md`). Two terms used throughout: - **Probe**: a minimal disposable artifact (a hook that dumps its environment, a script that captures stdin) run against the real pinned tool to observe what the tool actually does, instead of what its docs say. - **Negative control**: a deliberately broken input run through the same harness as the passing case, proving the test or probe CAN fail. A check that cannot fail proves nothing. ## Triggers - `probe the runtime contract` - `calibrate this guard` - `prove it empirically` - `add a negative control` - `docs versus reality audit` ## Recipe Selector | Situation | Recipe | Related skill | |-----------|--------|---------------| | You are about to rely on undocumented tool behavior (cwd, env vars, stdin format) | 1. Runtime-contract probe | `agent-harness-reference` for settled contracts | | You are shipping a detector, guard, ...

Details

Author
rjmurillo
Repository
rjmurillo/ai-agents
Created
7 months ago
Last Updated
today
Language
Markdown
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

ai-agents-research-methodology

How a hunch becomes an accepted result in this repo. Covers the evidence bar, hypothesis-predicts-numbers discipline, and the idea lifecycle from contradiction log through probe, eval baseline, ADR debate, calibrated gate, and guard-maturity monitoring. Use when you say `how do I prove this idea`, `run the idea lifecycle`, `what is the evidence bar`. Do NOT use for the open research programs (use ai-agents-research-frontier) or probe recipe depth (use ai-agents-empirical-probe-toolkit).

38 Updated today
rjmurillo
AI & Automation Solid

ai-agents-diagnostics-toolkit

Catalog of this repo's measurement instruments, each with command, current baseline, and interpretation guide. Covers skill size and description budgets, orphan-ref and golden-principles scans, drift gates as signals, guard telemetry and maturity tiers, coverage pins, and the eval harness. Use when you say `measure this`, `read the drift signal`, `check skill budgets`, `guard maturity report`. Do NOT use to fix what you measure (use `ai-agents-debugging-playbook`) or for evidence standards (use `ai-agents-validation-and-qa`).

38 Updated today
rjmurillo
AI & Automation Solid

ai-agents-portability-campaign

Execute cross-harness hook changes for Claude Code and GitHub Copilot CLI using the settled official contract, versioned probes, generators, and runtime tests. Use for `run the portability campaign`, `port hooks to a new harness`, `copilot hook timeout regression`, or `new copilot cli release, recheck the contract`. Do NOT use for fact lookup alone (use `agent-harness-reference`).

38 Updated today
rjmurillo