openai-compatible-embeddings-batchinglisted

# OpenAI-Compatible Embeddings: Batch Transparently or Fail Silently **Extracted:** 2026-05-23 **Context:** Any project using a custom adapter against an OpenAI-compatible embeddings endpoint (OpenRouter, vLLM gateways, Together, Anyscale, Groq, self-hosted TGI, etc.) with a corpus larger than a few dozen documents. ## Problem OpenAI-protocol `/embeddings` endpoints accept a list under `input` and return embeddings in `data[].embedding`. Per-request token limits vary by provider and underlying model — OpenRouter + `google/gemini-embedding-2-preview` chokes above ~500 small documents in one POST. **The dangerous failure shape:** rather than returning HTTP 4xx, OpenRouter returns **HTTP 200 with a non-standard error body that lacks the `data` key**. `response.raise_for_status()` passes. The next line — `[item["embedding"] for item in body["data"]]` — surfaces as a bare `KeyError: 'data'` deep inside LangChain or wherever the adapter is invoked. Zero signal about the actual cause. Worst case: the test suite that embedded 10 docs passes; production embed of 500+ docs crashes the first time a user opts in to the large corpus. ## Solution Three layers at the adapter level — the third is what makes it actually safe: 1. **Group by count + an approximate size budget.** A fixed document *count* per batch does NOT stay under the provider's per-request *token*/*byte* limit — 50 short docs and 50 long ones are very different payloads. Grouping by both a doc count and an approximat