algo-nlp-similarity

# Text Similarity ## Overview Text similarity measures how close two texts are in meaning or surface form. Lexical methods (Jaccard, cosine on TF-IDF) compare word overlap. Semantic methods (sentence embeddings) capture meaning even with different words. Choice depends on whether you need exact matching or meaning matching. ## When to Use **Trigger conditions:** - Finding similar or duplicate documents in a collection - Matching queries to FAQ answers or knowledge base entries - Detecting plagiarism or content reuse **When NOT to use:** - For topic-level grouping (use topic modeling / LDA) - For entity extraction from text (use NER) ## Algorithm ``` IRON LAW: Lexical Similarity ≠ Semantic Similarity "The car is fast" and "The automobile is speedy" have LOW lexical similarity (different words) but HIGH semantic similarity (same meaning). "Bank of the river" and "Bank account" have HIGH lexical similarity but LOW semantic similarity. Choose the method that matches your definition of "similar." ``` ### Phase 1: Input Validation Determine: similarity type needed (lexical or semantic), text preprocessing requirements, scale (pairwise vs all-pairs vs query-to-corpus). **Gate:** Texts preprocessed, method selected. ### Phase 2: Core Algorithm **Lexical methods:** - Jaccard: |A∩B| / |A∪B| on word sets - Cosine on TF-IDF vectors: cos(θ) = (A·B) / (|A|×|B|) **Semantic methods:** - Sentence embeddings: encode texts with sentence-transformers (all-MiniLM-L6-v2) - Cosine similar...