agent-observability-spec

# Agent Observability Spec Skill You can't fix what you didn't record. For LLM systems the unit of observability is the *trace* — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone. ## What This Skill Produces - A **trace schema**: per-request spans and the fields each must carry - **Metric definitions** across health, quality, cost, and behaviour — each with a threshold and owner - A **sampling and retention policy** that keeps cost sane and debugging possible - A **privacy note**: what logged content contains, who can see it, and how long it lives ## Required Inputs Ask for (if not already provided): - **The system's shape** — single LLM call, RAG pipeline, or multi-step tool-using agent - **Traffic volume and cost sensitivity** — full tracing at 10M req/day is a budget decision - **What "misbehaving" means here** — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?) - **Existing observability stack** (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it ## Trace Schema Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields: | Span | Must capture | |---|---| | **Request root** | request id, user/session (pseudonymous), feature + prompt version, model id, total tokens, total cost, latency, terminal status | | **Model call** | fu...