← ClaudeAtlas

replayable-audit-logslisted

Design pattern for observability-by-default in LLM agents — every feature that performs external I/O, calls an LLM, or makes non-deterministic/heuristic decisions ships a replayable append-only JSONL audit log in the same change. Use when adding or reviewing such a feature (the review question "which log answers why, and can we replay it offline?"), when designing a new audit record schema, when a recurring failure needs corpus-driven repair (replay harness, positive/negative ground truth, regression fixtures from real traffic), or when deciding how to store untrusted text in a log.
shimo4228/agent-observability-patterns · ★ 1 · AI & Automation · score 78
Install: claude install-skill shimo4228/agent-observability-patterns
# Replayable Audit Logs (observability by default) > Sibling pattern: an **instrument** reads across the whole store at query time > (distributions, calibration) — see > [`read-only-instruments`](../read-only-instruments/SKILL.md). The log is the > corpus; the instrument is one lens over it. A third sibling, > [`shadow-mode-validation`](../shadow-mode-validation/SKILL.md), uses a log of > this kind as the record of a candidate mechanism's would-be decisions. The pattern sits in the lineage of append-only logs and event sourcing: the record of what happened is written as an immutable sequence at the moment it happens, and every later analysis is a read over that sequence. Principle: **a feature with external I/O, LLM calls, or heuristic decisions ships its audit log in the same change** — because the corpus must predate the failure it will one day explain. Worked example: an agent that submits generated answers to an external verifier logged every attempt as an ordinary side effect for weeks; when its parser needed repair, hundreds of real logged challenges could be replayed offline against a zero-wrong hard gate, and the repair was validated against real traffic instead of synthetic cases. Ad-hoc logging added at investigation time can never provide that. ## Record schema checklist Design the record so the run can be **replayed offline**, not merely read: - [ ] **Raw input, recoverable** — the exact input the decision saw. Untrusted text (API responses, user/conte