cohere-performance-tuning

# Cohere Performance Tuning ## Overview Optimize Cohere API v2 performance through model selection, embedding batches, rerank pipelines, caching, and streaming for time-to-first-token. ## Prerequisites - `cohere-ai` SDK installed - Understanding of Cohere endpoints (Chat, Embed, Rerank) - Redis or in-memory cache (optional) ## Latency Benchmarks (Typical) | Operation | Model | P50 | P95 | |-----------|-------|-----|-----| | Chat (short) | `command-r7b-12-2024` | 500ms | 1.5s | | Chat (short) | `command-a-03-2025` | 800ms | 2.5s | | Chat (stream TTFT) | `command-a-03-2025` | 200ms | 600ms | | Embed (96 texts) | `embed-v4.0` | 150ms | 400ms | | Rerank (100 docs) | `rerank-v3.5` | 100ms | 300ms | | Classify (96 inputs) | `embed-english-v3.0` | 200ms | 500ms | ## Instructions ### Strategy 1: Model Selection by Latency Budget ```typescript // Use smaller models for latency-sensitive paths function selectModel(latencyBudgetMs: number): string { if (latencyBudgetMs < 1000) return 'command-r7b-12-2024'; // 7B, fastest if (latencyBudgetMs < 3000) return 'command-r-08-2024'; // Mid-tier return 'command-a-03-2025'; // Best quality } // Pair with maxTokens to control output length await cohere.chat({ model: selectModel(1500), messages: [{ role: 'user', content: query }], maxTokens: 200, // Shorter output = lower latency }); ``` ### Strategy 2: Streaming for Time-to-First-Token ```typescript // Non-streaming: user waits for ...