← ClaudeAtlas

lucene-retrieval-usagelisted

This skill should be used when indexing or searching a corpus with the pi-serini Lucene BM25 retriever, tuning BM25 k1/b parameters for long documents, handling missing Java 21 / pyserini dependencies, or when corpus scale demands a production-grade Lucene index.
josix/agentic-retrieval · ★ 0 · Data & Documents · score 72
Install: claude install-skill josix/agentic-retrieval
# Lucene Retrieval Usage (pi-serini) Lucene BM25 retrieval via `PiSeriniRetriever` (`retrieval/retrievers.py`), backed by Pyserini/Anserini — the reference lexical retriever from [pi-serini](https://github.com/justram/pi-serini) ("Rethinking Agentic Search with Pi-Serini: Is Lexical Retrieval Sufficient?"). `pi-serini` and `pyserini` are two different names, and neither is a typo for the other: `pi-serini` is the retrieval *strategy* from the paper above (and this plugin's registry key / `--retriever` value), while `pyserini` is the Castorini Python library — and this repo's install extra — that the strategy is implemented on top of. Wherever both appear on the same line below, the difference is deliberate. ## What it is `PiSeriniRetriever` builds an in-memory Lucene inverted index (`pyserini.index.lucene.LuceneIndexer`) over the corpus and queries it with `LuceneSearcher.set_bm25(k1, b)`. Pi-serini's thesis: a well-configured lexical (BM25) retriever, given **sufficient retrieval depth**, can be competitive with dense retrieval for agentic deep research — instead of closing the vocabulary gap with embeddings, retrieve deeper and let a capable LLM agent compensate by reading more candidates. Pi-serini's BrowseComp-Plus-tuned defaults push BM25 far from its usual settings: `k1=25` (vs. Lucene's usual ~0.9) nearly disables term-frequency saturation, letting repeated query terms keep accumulating score, and `b=1` applies full document-length normalization — tuned for long (