coreweave-observability

# CoreWeave Observability ## Overview CoreWeave runs GPU-intensive workloads on Kubernetes where hardware failures, memory exhaustion, and underutilization directly impact cost and reliability. Observability must cover DCGM GPU metrics, Kubernetes pod health, inference latency, and job completion rates. Proactive monitoring prevents wasted spend on idle GPUs and catches OOM conditions before they cascade. ## Key Metrics | Metric | Type | Target | Alert Threshold | |--------|------|--------|-----------------| | GPU utilization | Gauge | > 60% | < 20% for 30m | | GPU memory usage | Gauge | < 85% | > 95% for 5m | | Inference latency p99 | Histogram | < 200ms | > 500ms | | Job completion rate | Counter | > 99% | < 95% per hour | | Pod restart count | Counter | 0 | > 3 in 15m | | Node GPU temperature | Gauge | < 80C | > 85C for 10m | ## Instrumentation ```typescript async function trackInference(model: string, fn: () => Promise<any>) { const start = Date.now(); try { const result = await fn(); metrics.record('coreweave.inference.latency', Date.now() - start, { model, status: 'ok' }); metrics.increment('coreweave.inference.completed', { model }); return result; } catch (err) { metrics.increment('coreweave.inference.errors', { model, error: err.code }); throw err; } } ``` ## Health Check Dashboard ```typescript async function coreweaveHealth(): Promise<Record<string, string>> { const gpu = await queryPrometheus('avg(DCGM_FI_DEV_GPU_UTIL)'); ...