ai-distributed-training

# Distributed Training - Systems Performance Skill **Operational focus**: picking and implementing the right parallelism strategy, not the theory. Covers data parallelism through FSDP/ZeRO/tensor+pipeline parallelism, memory-efficient attention, mixed precision at scale, activation checkpointing, rented-GPU cost discipline, and reproducing GPT-2 124M as the canonical sanity check. Profile before you scale. Debug on the smallest GPU that fits. Stop the instance when done. ## ASCII Flow ```text single GPU (debug/prototype) └─ DDP: replicate model, all-reduce gradients — linear scale up to ~8 GPUs └─ FSDP2 / ZeRO: shard optimizer state, gradients, params across GPUs └─ tensor parallelism: split weight matrices across GPUs (intra-node) └─ pipeline parallelism: split layers across nodes (inter-node) └─ context parallelism: shard the sequence dim (long context) └─ expert parallelism: route MoE experts across GPUs (all-to-all) └─ N-D parallelism: DP + TP + PP + CP + EP (frontier MoE) profile-before-scale └─ nsys / torch.profiler → find bottleneck (compute? memory? dataloader?) └─ fix bottleneck at small scale, then scale ``` ## When to Use This Skill Activate when the user asks about: - Choosing between DDP, FSDP2, DeepSpeed ZeRO stages 1/2/3, or Megatron-LM - Training Mixture-of-Experts (MoE) models: expert parallelism, all-to-all, load balancing - OOM errors on multi-G...