clickhouse-observability

# ClickHouse Observability ## Overview Set up comprehensive monitoring for ClickHouse using built-in system tables, Prometheus integration, Grafana dashboards, and alerting rules. ## Prerequisites - ClickHouse instance with `system.*` table access - Prometheus (or compatible: Grafana Alloy, Victoria Metrics) - Grafana for dashboards - AlertManager or PagerDuty for alerts ## Instructions ### Step 1: Key Metrics from System Tables ```sql -- Real-time server health snapshot SELECT (SELECT count() FROM system.processes) AS running_queries, (SELECT value FROM system.metrics WHERE metric = 'MemoryTracking') AS memory_bytes, (SELECT value FROM system.metrics WHERE metric = 'Query') AS concurrent_queries, (SELECT count() FROM system.merges) AS active_merges, (SELECT value FROM system.asynchronous_metrics WHERE metric = 'Uptime') AS uptime_sec; -- Query throughput (last hour, per minute) SELECT toStartOfMinute(event_time) AS minute, count() AS queries, countIf(exception_code != 0) AS errors, round(avg(query_duration_ms)) AS avg_ms, round(quantile(0.95)(query_duration_ms)) AS p95_ms, formatReadableSize(sum(read_bytes)) AS total_read FROM system.query_log WHERE type IN ('QueryFinish', 'ExceptionWhileProcessing') AND event_time >= now() - INTERVAL 1 HOUR GROUP BY minute ORDER BY minute; -- Insert throughput (last hour) SELECT toStartOfMinute(event_time) AS minute, count() AS inserts, sum(written_rows) AS rows_written, ...