llm-integration

# LLM Integration ## Purpose Integrate a language model into a production application, where the API is slow, rate-limited, occasionally down, and billed per token — none of which the quickstart mentions. ## When to Use - Adding an LLM to a production application. - An LLM feature that is slow, expensive, or unreliable. - Handling rate limits, streaming, or provider failover. - Deciding where the model call belongs in the architecture. ## Capabilities - Streaming responses and partial rendering. - Retry, backoff, and rate-limit handling. - Timeouts and cancellation. - Prompt caching and response caching. - Multi-provider fallback. - Token accounting and cost control. ## Inputs - The feature, its latency budget, and its cost budget. - The provider's rate limits and their actual behavior under load. - Whether the output is user-facing (stream it) or machine-consumed (do not). ## Outputs - A client with retries, timeouts, and a circuit breaker. - Streaming where a user is waiting. - Cost and latency instrumented per call. ## Workflow 1. **Stream anything a human waits for** — A 12-second response that starts rendering at 400ms feels fast. The same response delivered at once feels broken. Streaming is a perceived-latency fix, not a throughput one. 2. **Handle rate limits properly** — Honor `Retry-After`. Exponential backoff with jitter. A retry storm against a rate-limited endpoint extends the outage. 3. **Set a timeout** — LLM calls can hang. An unbounded call holds ...