litellm-valkey

Solid

Run LiteLLM proxy multi-pod with Redis/Valkey (standalone, Sentinel, or Cluster) so rate limits, budgets, and router state are actually shared across pods. Centerpiece: LiteLLM's dominant failure mode is SILENT fallback from Redis to per-pod in-memory enforcement (≥6 independent code paths, log-level warning, no metric) — a fleet quietly enforces N× its configured limits. Covers the 4-way coordination-Redis resolution order (incl. the undocumented `general_settings.coordination_redis` block, first-class since v1.93.0, and the DB-persisted UI value that silently outranks the config file), Sentinel/Cluster support per subsystem with fix versions, the per-pod-vs-shared truth table with real staleness bounds, documented knobs that are no-ops, and the open-issue catalog (counter drift, connection churn, ssl_check_hostname).

AI & Automation 5 stars 1 forks Updated today MIT

Install

Quality Score: 80/100

Stars 20%

26

Recency 20%

100

Frontmatter 20%

70

Documentation 15%

100

Issue Health 10%

80

License 10%

100

Description 5%

100

Skill Content

# LiteLLM proxy + Redis/Valkey multi-pod — operator reference Target: operating LiteLLM proxy with 2+ replicas (typically the in-repo Helm chart) where Redis or Valkey is supposed to make rate limits, spend/budget enforcement, cooldowns, and locks fleet-wide. Grounded in source at `4d543245` (v1.95.0-dev, 2026-07-29; latest stable v1.94.0) plus a GitHub-issue sweep of the same date. LiteLLM releases weekly and fixes land fast — treat every claim as version-stamped, and re-verify on the deployed tag. Sibling skill: the proxy's management REST API (keys, teams, budgets semantics) is **`litellm-api`**. Migrating the Redis itself to Valkey is **`redis-to-valkey`**. The single most important thing to internalize: **when Redis fails — or is never wired in — LiteLLM does not fail. It silently enforces everything per-pod.** No 5xx, no metric, log-level `warning` at best. A fleet of N pods enforces N× every rate limit and lets budgets drift. Verifying that coordination is *actually* shared is the operator's job; nothing in the product surfaces the loss. Read `references/silent-degradation.md` first. ## Which Redis am I actually coordinating through? The coordination Redis (rate limits, spend counters, pod locks — NOT the response cache) is resolved in this precedence order (`proxy_server.py` `_init_coordination_redis` → `_init_cache`): 1. **`general_settings.coordination_redis`** in config.yaml — the only clean way to point coordination at its own Redis. First-class since **v1.9...

Details

Author: air-gapped
Repository: air-gapped/skills
Created: 4 months ago
Last Updated: today
Language: Python
License: MIT

Integrates with

Anthropic · AI Redis · Database Kubernetes · Infrastructure REST API · API

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

litellm-api

Administer a LiteLLM proxy via its management REST API (keys, teams, users, orgs, models, budgets, spend) — a 447-path control plane where ~63% of paths are undocumented, 91 are hidden from /openapi.json, and the OpenAPI schema documents wrong field names. Centerpiece: four "empty means UNLIMITED" footguns (`models: []`, `allowed_routes: []`, additive access groups, team-member `[]`) where the natural reading is the opposite. Covers the real auth model (no-master-key = fully open, admin-viewer default-allow on GETs, /v1/mcp/* RBAC bypass), budget semantics (can't unset, update≠reset, team spend hits personal budgets since ~v1.94), presence-sensitive updates that trigger spurious enterprise 403s, config-vs-DB precedence, and lazy-loaded families that 404 until warmed.

5 Updated today

DevOps & Infrastructure Solid

redis-to-valkey

Migrate Redis deployments (especially Bitnami Redis Helm charts in Sentinel HA mode) to Valkey on Kubernetes, including fully air-gapped clusters. Core knowledge: the RDB-version wall (Valkey replicates/loads only from Redis ≤ 7.2.x; Redis 7.4+ writes RDB v12 which Valkey rejects; Valkey 9 writes its own v80 — a one-way door), the two transfer layers (version-bound REPLICAOF/DUMP-RESTORE vs version-agnostic logical replay with RedisShake or rdb-cli), a side-by-side cutover runbook, Valkey chart selection (groundhog2k / CloudPirates / official valkey-io tradeoffs), Bitnami-redis→valkey values translation, consumer-app reconnection (Sentinel discovery, master-set names, frozen redis_version 7.2.4), Prometheus exporter continuity, air-gap tool/image mirroring, and Argo CD source rewiring away from charts.bitnami.com. Part of the bitnami-exit suite.

5 Updated today

AI & Automation Solid

vllm-deployment

Use this skill when authoring, reviewing, or fixing a vLLM Kubernetes manifest, Docker/Podman pod, or OpenShift ServingRuntime — even when the user does not say "vllm". Triggers on: lab cluster performance practices, cache mount + survival across pod restarts (/root/.cache, VLLM_CACHE_ROOT, TORCHINDUCTOR_CACHE_DIR, TRITON_CACHE_DIR, "do we have caches saved"), HF_TOKEN secret in pod env, liveness + readiness probe tuning (initialDelaySeconds, failureThreshold, "pod takes 12 min to boot"), serve_args review, --enforce-eager rationale, MoE deployment ("ep2 dp2", --enable-expert-parallel, expert-parallel sizing), TP/PP sizing, ConfigMap parser-plugin mount, image tag selection, cold-boot reduction, multi-node LWS + Ray, control planes (llm-d, production-stack, AIBrix, NVIDIA Dynamo, KServe), KEDA autoscaling, GAIE routing, disaggregated prefill/decode (Nixl/Mooncake/LMCache/MORI-IO), RHAIIS on OpenShift (SCC, arbitrary UID, Routes 60s, ModelCar, air-gapped). Lead with operator intent, not vendor names.

5 Updated today