nemo-mbridge-perf-cpu-offloadinglisted

# CPU Offloading ## References - Stable docs: @docs/training/cpu-offloading.md - Structured metadata: @skills/nemo-mbridge-perf-cpu-offloading/card.yaml ## What It Is Two independent mechanisms to move data from GPU to CPU memory: | Mechanism | Config namespace | What gets offloaded | PP restriction | |---|---|---|---| | Activation offloading | `model.cpu_offloading*` | Activations (and optionally weights) per transformer layer | PP must be 1 | | Optimizer offloading | `optimizer.optimizer_cpu_offload` | Adam optimizer states (momentum + variance) via `HybridDeviceOptimizer` | None | ## Quick Decision | Situation | Recommendation | |---|---| | Large MoE model (30B+), needs PP > 1 | Optimizer offloading — activation offloading is blocked by PP=1 | | Small/medium model, PP=1 fits, activation memory dominates | Activation offloading | | Want tunable memory-speed tradeoff | Optimizer offloading with fractional `optimizer_offload_fraction` | | Throughput is top priority | Don't enable — offloading always adds overhead | | CUDA graphs are needed | Only optimizer offloading — activation offloading is incompatible | | Memory pressure is moderate | Optimizer offload at 25–50% fraction for best efficiency | ## Enablement ### Optimizer CPU offloading (recommended for large models) ```python cfg.optimizer.optimizer_cpu_offload = True cfg.optimizer.optimizer_offload_fraction = 1.0 cfg.optimizer.overlap_cpu_optimizer_d2h_h2d = True ``` CLI overrides: ```bash optimizer.optimizer