langfuse-incident-runbook

# Langfuse Incident Runbook ## Overview Step-by-step procedures for Langfuse-related incidents, from initial triage (2 min) through resolution and post-incident review. Your application should work without Langfuse -- these procedures focus on restoring observability. ## Severity Classification | Severity | Description | Response Time | Example | |----------|-------------|---------------|---------| | P1 | Application impacted by tracing | 15 min | SDK throwing unhandled errors, blocking requests | | P2 | Traces not appearing, no app impact | 1 hour | Missing observability data | | P3 | Degraded performance from tracing | 4 hours | High latency from flush backlog | | P4 | Minor issues | 24 hours | Occasional missing traces | ## Instructions ### Step 1: Initial Assessment (2 Minutes) ```bash set -euo pipefail echo "=== Langfuse Incident Triage ===" echo "Time: $(date -u)" # 1. Check Langfuse cloud status echo -n "Status page: " curl -s -o /dev/null -w "%{http_code}" https://status.langfuse.com || echo "UNREACHABLE" echo "" # 2. Test API connectivity HOST="${LANGFUSE_BASE_URL:-${LANGFUSE_HOST:-https://cloud.langfuse.com}}" echo -n "API health: " curl -s -o /dev/null -w "%{http_code} (%{time_total}s)" "$HOST/api/public/health" || echo "FAILED" echo "" # 3. Test auth if [ -n "${LANGFUSE_PUBLIC_KEY:-}" ] && [ -n "${LANGFUSE_SECRET_KEY:-}" ]; then AUTH=$(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64) echo -n "Auth test: " curl -s -o /dev/null -w "%{http...