experiment-audit

Featured

Audit experiment integrity before claiming results. Uses cross-model review (external reviewer backend) to check for fake ground truth, score normalization fraud, phantom results, and insufficient scope. Use when user says "审计实验", "check experiment integrity", "audit results", "实验诚实度", or after experiments complete before writing claims.

AI & Automation 14,964 stars 1313 forks Updated today MIT

Install

View on GitHub

Quality Score: 96/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Experiment Audit: Cross-Model Integrity Verification > 🔒 **Do not wrap this skill in `/loop`, `/schedule`, or `CronCreate`.** It is > verdict-bearing — it judges experiment integrity. Re-running that verdict on a > timer adds no new signal, and a loop that accepts its own output to decide > when to stop crosses into self-acquittal (`acceptance-gate.md`). Schedule the > *external wait that precedes it* — experiments done → then audit **once**. See > [`shared-references/external-cadence.md`](../shared-references/external-cadence.md). Audit experiment integrity for: **$ARGUMENTS** ## Why This Exists LLM agents can produce fraudulent experimental results through: 1. **Fake ground truth** — creating synthetic "reference" from model outputs, then reporting high agreement as performance 2. **Score normalization** — dividing metrics by the model's own max to get 0.99+ 3. **Phantom results** — claiming numbers from files that don't exist or functions never called 4. **Insufficient scope** — reporting 2-scene pilots as "comprehensive evaluation" These are NOT intentional deception — they are failure modes of optimizing agents that lack integrity constraints. This skill adds that constraint. ## Core Principle **The executor collects file paths. The external reviewer backend reads code and judges integrity. The executor does NOT participate in integrity judgment.** This follows `shared-references/reviewer-independence.md` and `shared-references/experiment-integrity.md`. ## Co...

Details

Author
wanshuiyin
Repository
wanshuiyin/Auto-claude-code-research-in-sleep
Created
5 months ago
Last Updated
today
Language
Python
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Featured

experiment-forensics

Audit experiment integrity against the evidence ledger. At L2 (repo + result files present) a fresh cross-model reviewer reads the eval code line-by-line for fake/derived ground truth, score self-normalization, phantom results (a paper number with no backing file/key), dead/uncalled metric code, verified-scope inflation, method-described ≠ method-evaluated drift, synthesized-looking results, placeholder/fake data still wired into a released result, code-output ≠ reported-number mismatch, and missing reproducibility artifacts (an empirical/agent/LLM paper shipping neither code nor the prompts/configs its results need) — every finding span-anchored to a ledger claim_id. At L0/L1 (PDF / source only) the same patterns are surfaced as info-level 'could-not-verify' signals where the ledger gives an anchor (observability_level_required:2) — NEVER a fraud verdict from a PDF. The reviewer PROPOSES findings; tools/adjudicate_findings.py computes the verdict. Detect-only. Triggers: "experiment forensics", "audit the res

139 Updated 3 days ago
wanshuiyin
AI & Automation Listed

experiment-audit

Use this skill for scientific and ML-research reasoning work — evaluating experimental claims, auditing training runs or ablations, checking whether a statistical claim holds up, assessing reproducibility, reconciling contradictory results, reviewing a paper's methodology or results section, writing reviewer-style feedback, or producing a structured research report. This is a scientific reasoning discipline, not just a tool wrapper, so it applies even with no live data source — e.g. reviewing a pasted table of results, sanity-checking a claimed effect size, or evaluating an ablation described in prose. Trigger on phrasing like "did I mess up this experiment," "is this result real," "why did my loss/reward do X," "which run is better," "is this ablation confounded," "review this paper's claims," "write reviewer feedback," "is this reproducible," "what should I conclude from this," or "write up these results." When the user's data lives in Weights & Biases, this skill also covers the experiment-audit-mcp integr

4 Updated 1 months ago
SreeDharshan-GJ
AI & Automation Listed

paper-claim-audit

Zero-context verification that every number, comparison, and scope claim in the paper matches raw result files. Uses a fresh cross-model reviewer with NO prior context to prevent confirmation bias. Use when user says "审查论文数据", "check paper claims", "verify numbers", "论文数字核对", or before submission to ensure paper-to-evidence fidelity. Do not use for code/data/formula verification (use paper-verification).

2 Updated 3 weeks ago
wookat