instrument-llm-observabilitylisted

Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral OpenTelemetry unless a specific platform is already in use.
ContextJet-ai/awesome-llm-observability · ★ 26 · AI & Automation · score 72

Install: claude install-skill ContextJet-ai/awesome-llm-observability

# Instrument an LLM app for observability Add production-grade tracing to an LLM/agent app so every run is inspectable: prompts, tool calls, retrievals, token usage, latency, and cost per span. ## Decide the approach first (2 questions) 1. **Is a backend already chosen?** (Langfuse, Arize Phoenix, Comet Opik, Datadog, Grafana…). If yes, use its SDK/auto-instrumentation. If no, default to **OpenTelemetry GenAI semantic conventions** so traces export to *any* OTel backend and you avoid lock-in. 2. **Lowest-friction path available?** If the app calls LLMs through a **gateway/proxy** (LiteLLM, Helicone, Portkey), you may get tracing + cost with *zero code* by changing the base URL. Check that before adding an SDK. ## Vendor-neutral path (recommended default): OpenTelemetry Emit `gen_ai.*` spans via an OTel instrumentation library, then point at any collector. - Use **OpenLLMetry** (`traceloop-sdk`) or **OpenInference** (Arize) - both emit OTel-compatible LLM spans. - Minimal shape: 1. Install the instrumentation lib + an OTLP exporter. 2. Initialize once at startup (set service name + OTLP endpoint via `OTEL_EXPORTER_OTLP_ENDPOINT`). 3. Auto-instrument the LLM SDK (OpenAI/Anthropic/etc.); wrap custom agent steps in manual spans (`gen_ai.operation.name`, tool spans). - Result: spans carry model, prompt/completion, token counts, latency; cost is derived downstream. See `references/opentelemetry.md` for the exact span attributes to set and a copy-paste init. ## Platfor