ml-architecture-datalisted
Install: claude install-skill Lu1sDV/skillsmd
# ML Architecture, Tokenization & Data Processing
Three capabilities, one reference each. Open only the reference the task needs;
do not preload all.
| Task / Tool | Read |
|---|---|
| Model architectures & pretraining/fine-tuning: LitGPT, TorchTitan (4D parallelism, FSDP2, Float8), nanoGPT, Mamba, RWKV | [Architectures reference](references/architectures.md) |
| Subword tokenization: Hugging Face `tokenizers` (BPE, WordPiece, Unigram, offsets) and `sentencepiece` (raw Unicode, CJK, subword regularization) | [Tokenization reference](references/tokenization.md) |
| Data curation & ETL: NeMo Curator (GPU dedup, heuristic/classifier filtering, PII, multimodal) and Ray Data (streaming `map_batches`, Ray Train sharding) | [Data processing reference](references/data-processing.md) |
## What each covers
- **Architectures** — Decision matrix and workflows for LitGPT (20+ models, LoRA/QLoRA, GGUF), TorchTitan (PyTorch-native FSDP2, TP/PP/CP, H100 Float8), nanoGPT (~300-line educational GPT-2), Mamba (`O(n)` selective SSM without KV cache), and RWKV (`O(1)` inference memory RNN+Transformer hybrid).
- **Tokenization** — Training custom BPE/WordPiece/Unigram vocabularies, 4-stage normalization/pre-tokenization/post-processing pipelines, span alignment tracking, `PreTrainedTokenizerFast` wrapping, and SentencePiece CJK/multilingual setup.
- **Data processing** — Ordered LLM curation pipelines (heuristics -> language ID -> exact/fuzzy/semantic dedup -> PII redaction -> GPU classifiers)