langchain-data-handling

# LangChain Data Handling: RAG & Document Processing ## Overview Build Retrieval-Augmented Generation (RAG) pipelines: load documents, split into chunks, embed with OpenAI/Cohere, store in vector databases (FAISS, Chroma, Pinecone), and query with retrieval chains. ## Prerequisites - `@langchain/core`, `@langchain/openai` installed - For vector stores: `npm install @langchain/community` (FAISS) or `npm install @langchain/pinecone @pinecone-database/pinecone` ## Step 1: Document Loaders ```typescript import { TextLoader } from "langchain/document_loaders/fs/text"; import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf"; import { DirectoryLoader } from "langchain/document_loaders/fs/directory"; import { CSVLoader } from "@langchain/community/document_loaders/fs/csv"; // Load a single file const textDocs = await new TextLoader("./data/readme.md").load(); const pdfDocs = await new PDFLoader("./data/manual.pdf").load(); // Load entire directory with type-based routing const dirLoader = new DirectoryLoader("./data/", { ".txt": (path) => new TextLoader(path), ".pdf": (path) => new PDFLoader(path), ".csv": (path) => new CSVLoader(path), }); const allDocs = await dirLoader.load(); console.log(`Loaded ${allDocs.length} documents`); ``` ## Step 2: Text Splitting ```typescript import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters"; const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, // max chars per chunk c...