vastai-observability

# Vast.ai Observability ## Overview Monitor Vast.ai GPU instance health, utilization, and costs. Key metrics: GPU utilization (idle GPUs waste $0.20-$4.00/hr), instance uptime, training progress, cost accumulation, and spot preemption events. ## Prerequisites - Vast.ai account with active instances - `vastai` CLI installed - Optional: Prometheus, Grafana, or Datadog for dashboarding ## Instructions ### Step 1: Instance Metrics Collector ```python import subprocess, json, time from datetime import datetime class VastMetricsCollector: def __init__(self, output_file="vast_metrics.jsonl"): self.output_file = output_file def collect(self): result = subprocess.run( ["vastai", "show", "instances", "--raw"], capture_output=True, text=True) instances = json.loads(result.stdout) metrics = { "timestamp": datetime.utcnow().isoformat(), "total_instances": len(instances), "running": 0, "total_hourly_cost": 0, "instances": [], } for inst in instances: status = inst.get("actual_status", "unknown") dph = inst.get("dph_total", 0) if status == "running": metrics["running"] += 1 metrics["total_hourly_cost"] += dph metrics["instances"].append({ "id": inst["id"], "gpu": inst.get("gpu_name"), "status": status, "dph": dph,...