← ClaudeAtlas

ml-architecture-datalisted

Use when choosing, configuring, or running LLM model architectures (LitGPT, TorchTitan, nanoGPT, Mamba, RWKV), training or integrating subword tokenizers (Hugging Face Tokenizers, SentencePiece), or building large-scale data curation and streaming ETL pipelines (NeMo Curator, Ray Data). Triggers on 'litgpt', 'torchtitan', 'nanogpt', 'mamba', 'rwkv', 'tokenizers', 'sentencepiece', 'nemo-curator', 'ray-data', 'FSDP2', 'MinHash deduplication', or 'BPE/Unigram'.
Lu1sDV/skillsmd · ★ 2 · AI & Automation · score 63
Install: claude install-skill Lu1sDV/skillsmd
# ML Architecture, Tokenization & Data Processing Three capabilities, one reference each. Open only the reference the task needs; do not preload all. | Task / Tool | Read | |---|---| | Model architectures & pretraining/fine-tuning: LitGPT, TorchTitan (4D parallelism, FSDP2, Float8), nanoGPT, Mamba, RWKV | [Architectures reference](references/architectures.md) | | Subword tokenization: Hugging Face `tokenizers` (BPE, WordPiece, Unigram, offsets) and `sentencepiece` (raw Unicode, CJK, subword regularization) | [Tokenization reference](references/tokenization.md) | | Data curation & ETL: NeMo Curator (GPU dedup, heuristic/classifier filtering, PII, multimodal) and Ray Data (streaming `map_batches`, Ray Train sharding) | [Data processing reference](references/data-processing.md) | ## What each covers - **Architectures** — Decision matrix and workflows for LitGPT (20+ models, LoRA/QLoRA, GGUF), TorchTitan (PyTorch-native FSDP2, TP/PP/CP, H100 Float8), nanoGPT (~300-line educational GPT-2), Mamba (`O(n)` selective SSM without KV cache), and RWKV (`O(1)` inference memory RNN+Transformer hybrid). - **Tokenization** — Training custom BPE/WordPiece/Unigram vocabularies, 4-stage normalization/pre-tokenization/post-processing pipelines, span alignment tracking, `PreTrainedTokenizerFast` wrapping, and SentencePiece CJK/multilingual setup. - **Data processing** — Ordered LLM curation pipelines (heuristics -> language ID -> exact/fuzzy/semantic dedup -> PII redaction -> GPU classifiers)