chunking-strategy-optimizerlisted

Chunking is the highest-leverage, most-overlooked knob in retrieval: if the right passage never lands in a single chunk, no reranker or bigger model recovers it. This skill replaces "512 tokens with 50 overlap, because that's what the tutorial said" with a measured choice — sweep candidate strategies over a fixed eval set and pick the one that actually retrieves the answers. ## When to use this skill - Standing up retrieval for a new corpus and you need a defensible chunking default. - RAG answers miss content you can see exists in the source documents. - Deciding chunk size, overlap, or strategy (token vs. sentence vs. recursive vs. semantic). - Migrating embedding models and want to re-confirm chunking still holds up. ## Instructions 1. **Build a retrieval eval set first.** Collect 20–50 real questions and, for each, the passage(s) that contain the answer (the "gold" spans). Hand-label if needed — even 20 cases beat eyeballing. This set is the ground truth every configuration is scored against; freeze it. 2. **Define the candidate configurations.** A small grid, not a search of everything: 2–3 strategies (e.g. recursive, sentence, semantic) × 2–3 sizes (e.g. 256 / 512 / 1024 tokens) × overlap (0 / 10–15%). Hold the embedding model and retriever fixed so chunking is the only variable. 3. **Run each configuration end to end.** For each config: chunk the corpus (e.g. with [Chonkie](/tools/chonkie)), embed the chunks with the fixed model, index them, and run the eval querie