together-reference-architecture

# Together AI Reference Architecture ## Overview Production architecture for AI inference, fine-tuning, and batch processing with Together AI's OpenAI-compatible API. Designed for teams routing requests across 100+ open-source models (Llama, Mixtral, Qwen, FLUX) with intelligent model selection, response caching, fine-tune pipeline management, and cost optimization via batch inference at 50% discount. Key design drivers: model routing for cost/quality tradeoffs, inference caching for repeated queries, fine-tune lifecycle management, and graceful degradation across model providers. ## Architecture Diagram ``` Application ──→ Model Router ──→ Cache (Redis) ──→ Together API (v1) ↓ /chat/completions Queue (Bull) ──→ Batch Worker /completions ↓ /images/generations Fine-Tune Manager ──→ Together API /fine-tunes ↓ /models Cost Tracker ──→ Analytics Dashboard ``` ## Service Layer ```typescript class InferenceService { constructor(private together: TogetherClient, private cache: CacheLayer, private router: ModelRouter) {} async complete(request: InferenceRequest): Promise<InferenceResponse> { const model = this.router.selectModel(request.task, request.priority); const cacheKey = `inference:${model}:${this.hashPrompt(request.prompt)}`; const cached = await thi...