rag-hybrid-search

# rag-hybrid-search Implement hybrid search combining semantic vector retrieval with keyword-based BM25 search for improved RAG pipeline accuracy and recall. ## Overview Hybrid search addresses the limitations of pure semantic or pure keyword search: - Semantic search excels at conceptual similarity but may miss exact matches - Keyword search finds exact terms but lacks semantic understanding - Hybrid combines both for superior retrieval performance ## Capabilities ### Search Strategies - Dense vector semantic search (embeddings) - Sparse vector keyword search (BM25, TF-IDF) - Hybrid fusion with configurable weighting - Reciprocal Rank Fusion (RRF) combination ### Retrieval Configuration - Configure embedding models for dense search - Tune BM25 parameters (k1, b values) - Set retrieval limits and thresholds - Apply metadata filtering ### Ranking & Reranking - Score normalization across search types - Weighted score fusion - Cross-encoder reranking - MMR (Maximum Marginal Relevance) diversity ### Index Management - Create and update hybrid indexes - Batch indexing with progress tracking - Index optimization and maintenance - Multi-index federation ## Usage ### Basic Hybrid Search with LangChain ```python from langchain_community.retrievers import BM25Retriever from langchain_community.vectorstores import Chroma from langchain.retrievers import EnsembleRetriever from langchain_openai import OpenAIEmbeddings # Create documents docs = [...] # Your document chunks # ...