distributed-tracing

# Skill: Distributed Tracing > **Expertise:** OpenTelemetry SDK, auto-instrumentation, Tempo/Jaeger, trace-log correlation, sampling strategies. ## When to load When adding tracing to a service, debugging slow distributed transactions, or setting up trace → log → metric correlation. ## End-to-End Setup Workflow 1. **Deploy collector** — configure and deploy the OTel Collector as a DaemonSet (see config below) 2. **Instrument service** — add SDK initialization and auto-instrumentation for your framework (Python/Go examples below) 3. **Verify traces** — confirm traces appear in Tempo/Jaeger: `curl -s http://tempo:3200/api/search?q={}&limit=5` 4. **Add log correlation** — inject `trace_id` and `span_id` into log lines for Loki/Grafana linkage 5. **Validate linkage** — click a trace in Grafana → Explore → verify it links to the corresponding log entries 6. **Tune sampling** — apply tail-based sampling policies for errors and slow traces (see strategy table) ## OpenTelemetry Collector (K8s DaemonSet) ```yaml # otel-collector-config.yaml receivers: otlp: protocols: grpc: { endpoint: "0.0.0.0:4317" } http: { endpoint: "0.0.0.0:4318" } processors: batch: timeout: 1s send_batch_size: 1000 memory_limiter: check_interval: 1s limit_mib: 400 # Tail-based sampling — sample 100% of error/slow traces tail_sampling: decision_wait: 10s policies: - name: errors-policy type: status_code status_code: { status_codes: ...