← ClaudeAtlas

mempalace-hnsw-corruption-fixlisted

Diagnose and fix MemPalace HNSW index corruption where link_lists.bin grows to hundreds of GB due to chromadb-hnswlib 1.5.x type-confusion in Rust bindings. Use when link_lists.bin is abnormally large, mine crashes with OOM, or header.bin shows astronomical element counts.
williamblair333/Uncle-J-s-Refinery · ★ 3 · AI & Automation · score 73
Install: claude install-skill williamblair333/Uncle-J-s-Refinery
## When to use - `~/.mempalace/palace/*/link_lists.bin` is abnormally large (GB+ range) - `mempalace mine` crashes with OOM or hangs - `header.bin` shows `max_elements` or `cur_element_count` in the trillions - Mine log shows repeated runs each making the file larger - SQLite drawer count diverges from HNSW `cur_element_count` ## Root cause chromadb 1.5.x defaults to `RustBindingsAPI`. The Rust HNSW binding has a type-confusion bug: `element_levels_[i]` is written as float but read as int32. Every `updatePoint` call (upsert of an existing item) triggers it, producing ~1 billion link-list sizes per node. **Critical:** Pinning `chromadb==1.5.8` is NOT sufficient. Without `chroma-hnswlib` installed, chromadb has no Python hnswlib fallback and always uses the broken Rust path. The repair itself calls `updatePoint` on existing items, immediately re-corrupting the fresh HNSW files. **Full fix requires both:** 1. `chroma-hnswlib==0.7.6` installed (provides the `hnswlib` Python module) 2. `CHROMA_API_IMPL=chromadb.api.segment.SegmentAPI` set in every entry point ## Step 1 — Install the Python hnswlib package ```bash # In the mempalace venv: uv pip install "chroma-hnswlib==0.7.6" # Verify it's present: python3 -c "import hnswlib; print(hnswlib.__file__)" ``` Also add to `pyproject.toml` under `[tool.uv.override-dependencies]` and `[project.dependencies]` so `uv sync` keeps it. ## Step 2 — Set CHROMA_API_IMPL in ALL entry points Every process that touches chromadb must expor