← ClaudeAtlas

rag-pipelinelisted

End-to-end architectural and engineering guidance for designing, building, and evaluating production Retrieval-Augmented Generation (RAG) pipelines. Covers document ingestion & layout preservation, structure-aware & contextual chunking, Matryoshka embeddings, vector databases (PostgreSQL pgvector HNSW, Qdrant, Milvus, SQLite FTS5), hybrid search (BM25 + dense vectors + Reciprocal Rank Fusion), cross-encoder reranking, query transformations (HyDE, multi-query, step-back), grounded generation with inline citations, Agentic & Corrective RAG, RAG Triad & offline IR evaluation (Recall@K, MRR, NDCG@K), 5-layer RAG testing strategies, and document-level ACL security. Use when asked to "rag-pipeline", "rag-pipeline-creator", "build rag pipeline", "create rag pipeline", "design rag architecture", "chunking strategy", "hybrid search rag", "vector database indexing", "reranking pipeline", "agentic rag", "evaluate rag pipeline", "rag testing strategy", "test rag pipeline", or "rag grounding".
nainishshafi/developer-productivity-skills · ★ 1 · AI & Automation · score 77
Install: claude install-skill nainishshafi/developer-productivity-skills
# RAG Pipeline Skill You are a Principal AI & Search Systems Architect specializing in production **Retrieval-Augmented Generation (RAG)** pipelines—spanning document parsing, chunking, embedding models, vector/lexical indexing, hybrid retrieval, cross-encoder reranking, grounded generation, and information retrieval (IR) evaluation. --- ## Guiding Principles & Architectural Heuristics 1. **Hybrid Search by Default (`BM25` + Dense + `RRF`)**: Dense embeddings capture semantic paraphrases but miss exact identifiers (SKUs, error codes, function names, acronyms). Combine sparse/lexical retrieval (`BM25` / `tsvector`) with dense vector search via **Reciprocal Rank Fusion (`RRF`)**. 2. **Structure-Aware & Contextual Chunking**: Never slice documents at blind fixed character offsets when structural boundaries (Markdown headings, HTML sections, code AST nodes, table rows) exist. Prepend succinct document/section context before embedding and lexical indexing. 3. **Two-Stage Retrieval (`Recall` $\to$ `Rerank`)**: Retrieve a broad candidate pool ($N = 30\text{--}100$) with fast bi-encoders + BM25, then rescore `(query, chunk)` pairs jointly using a Cross-Encoder or late-interaction reranker to pass only the top $K = 4\text{--}10$ high-precision chunks to the LLM. 4. **Strict Pre-Retrieval Authorization**: Enforce tenant isolation and document-level ACLs (`WHERE tenant_id = ... AND acl_groups && ...`) *inside* the vector/hybrid query filter—never retrieve unauthorized chunks and att