ai-scaling-laws

# AI Scaling Laws — Compute-Optimal Sizing Skill **Functional reference** for pre-training researchers and engineers who need to reason cold about compute, token, and parameter tradeoffs. Covers Kaplan et al. (2020), Chinchilla / Hoffmann et al. (2022), GPT-3 sizing, over-training for inference efficiency, and the mechanics of budget allocation for a from-scratch run. This is a standard interview probe. Know the key ratios and be ready to work through a concrete sizing calculation without a lookup. ## Quick Reference | Concept | Formula / Heuristic | Notes | |---------|---------------------|-------| | Compute budget | C ≈ 6 N D | N = non-embedding params, D = training tokens; approximate, constant ≈6 accounts for forward + backward | | Chinchilla-optimal ratio | D ≈ 20 × N | From Hoffmann et al. 2022; holds compute constant | | Kaplan (2020) ratio | D ≈ 1.7–2 × N (roughly) | Pre-Chinchilla; model-heavy. Difference from Chinchilla is *methodological* (FLOP counting, warmup, optimizer tuning), not simply "wrong" — see post-Chinchilla ref | | Optimal N given C | N* ≈ (C / 120)^0.5 | Approximate; from Chinchilla Table A3 | | Optimal D given C | D* ≈ (C / 0.3)^0.5 | Paired with above; verify against Hoffmann et al. Table A3 numbers | | Over-training (Llama-style) | D ≫ 20 × N | Trades higher training loss for cheaper inference; standard for deployed open models. Llama 3 8B: 15T tokens ≈ 1,875 tok/param (dense example). Llama 4 (2025) is the current MoE example — apply the rati...