chunking-strategy

# Chunking Strategy for RAG Systems ## Overview Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics. ## When to Use Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality. ## Instructions ### Choose Chunking Strategy Select based on document type and use case: 1. **Fixed-Size Chunking** (Level 1) - Use for simple documents without clear structure - Start with 512 tokens and 10-20% overlap - Adjust: 256 for factoid queries, 1024 for analytical 2. **Recursive Character Chunking** (Level 2) - Use for documents with structural boundaries - Hierarchical separators: paragraphs → sentences → words - Customize for document types (HTML, Markdown, JSON) 3. **Structure-Aware Chunking** (Level 3) - Use for structured content (Markdown, code, tables, PDFs) - Preserve semantic units: functions, sections, table blocks - Validate structure preservation post-split 4. **Semantic Chunking** (Level 4) - Use for complex documents with thematic shifts - Embedding-based boundary detection with 0.8 similarity threshold - Buffer size: 3-5 sentences 5. **Advanced Methods** (Level 5) - Late Chunking for long-context models - Contextual Retrieval for high-precision require...