openrouter-load-balancing

# OpenRouter Load Balancing ## Overview A single OpenRouter API key has rate limits (requests/minute and tokens/minute). To scale beyond those limits, distribute requests across multiple keys. OpenRouter also provides server-side load balancing via provider routing and the `:nitro` variant for low-latency inference. This skill covers multi-key rotation, health-based routing, circuit breakers, and concurrent request patterns. ## Multi-Key Round Robin ```python import os, itertools, time, logging from openai import OpenAI, RateLimitError from dataclasses import dataclass, field log = logging.getLogger("openrouter.lb") @dataclass class KeyPool: """Round-robin API key pool with health tracking.""" keys: list[str] _cycle: itertools.cycle = field(init=False, repr=False) _health: dict[str, dict] = field(init=False, default_factory=dict) def __post_init__(self): self._cycle = itertools.cycle(self.keys) self._health = {k: {"errors": 0, "last_error": 0, "healthy": True} for k in self.keys} def next_key(self) -> str: """Get next healthy key.""" attempts = 0 while attempts < len(self.keys): key = next(self._cycle) h = self._health[key] # Recover after 60s cooldown if not h["healthy"] and time.time() - h["last_error"] > 60: h["healthy"] = True h["errors"] = 0 if h["healthy"]: return key attempts += 1...