harness-firstlisted

# Harness first A model is an engine. Most "model problems" are a missing harness: nothing defines a correct output, nothing checks changes, nothing caps spend, the data has no agreed definitions, actions have no approval, and there are no traces to find the bad call. Audit that before touching the model. ## 1. Reproduce the symptom from evidence, not vibes - Read the agent code the symptom touches: the loop, tool definitions, prompt assembly, data access, side-effecting tools. - Read whatever logs/traces/transcripts exist. Aggregate them (per conversation, per tool, per error): who spends the tokens, which calls fail, which answers are wrong. A few outliers usually dominate. - State the root cause as a mechanism with evidence (file:line, log ids, numbers you computed). "The model is bad" is not a mechanism. Common mechanisms: - **Token burn:** loop with no max-iterations, retry on a deterministic error, full history or a huge tool result resent every turn, no caching of stable context. - **"Hallucination":** the agent faithfully used an ambiguous or wrong source (two tables/fields with the same name and different meaning, stale doc). Recompute both numbers yourself to prove it. - **Brittleness:** prompt changes shipped with no regression set; behavior that only the author checked by hand. ## 2. Score the six harness parts For each, mark **present / partial / missing** with the evidence you found: | Part | Present means | |---|---| | Golden set | ~20+ cases with expect