ai-pretraining

# Pretraining From Scratch **Domain**: building a transformer/GPT and a BPE tokenizer from first principles — the from-first-principles training-layer competency. Does NOT cover applications-layer fine-tuning, RLHF, or inference optimization; those belong to sibling skills. Canonical teachers: Karpathy "Neural Networks: Zero to Hero" (micrograd → makemore → "Let's build GPT" → "Let's build the GPT Tokenizer" → "Let's reproduce GPT-2"), Karpathy nanochat (full-stack from-scratch successor to nanoGPT, 2025), Raschka "Build a Large Language Model From Scratch", nanoGPT, minbpe, "Attention Is All You Need". GPT-2 is the pedagogical spine here — the right thing to build *first*. The 2026 from-scratch baseline then swaps four components onto that spine (RoPE, RMSNorm, SwiGLU, GQA) and runs attention through FlashAttention/SDPA; see [Modern Architecture Deltas](references/modern-architecture-deltas.md). ## ASCII Flow ```text Raw text corpus | v BPE Tokenizer (byte-level merges, vocab, encode/decode) | v Token IDs -> Embedding table (vocab_size x n_embd) | v + Positional Embedding (learned, shape: block_size x n_embd) | v Transformer Block x N ├── LayerNorm (pre-norm placement in GPT-2 style) ├── Multi-Head Self-Attention (causal mask, k/q/v projections) ├── Residual connection ├── LayerNorm ├── FFN (Linear -> GELU -> Linear, 4x expansion) └── Residual connection | v Final LayerNorm | v LM Head (Linear, n_embd -> vocab_size, weight-tied to emb...