nemo-mbridge-resiliencylisted

# Resiliency Stable docs: @docs/training/resiliency.md, @docs/training/checkpointing.md Card: @skills/nemo-mbridge-resiliency/card.yaml ## Enablement ### Fault tolerance (Slurm only) #### Option 1: NeMo Run plugin (recommended) ```python from megatron.bridge.recipes.run_plugins import FaultTolerancePlugin import nemo_run as run task = run.Script(...) run_plugins = [ FaultTolerancePlugin( enable_ft_package=True, calc_ft_timeouts=True, num_in_job_restarts=3, num_job_retries_on_failure=2, initial_rank_heartbeat_timeout=1800, rank_heartbeat_timeout=300, ) ] run.run(task, plugins=run_plugins, executor=executor) ``` | Plugin parameter | Default | Description | |---|---|---| | `num_in_job_restarts` | 3 | Max restarts within same job | | `num_job_retries_on_failure` | 2 | Max new job launches on failure | | `initial_rank_heartbeat_timeout` | 1800 | First heartbeat timeout (seconds) | | `rank_heartbeat_timeout` | 300 | Subsequent heartbeat timeout (seconds) | #### Option 2: Direct config + ft_launcher ```python from megatron.bridge.training.config import FaultToleranceConfig cfg.ft = FaultToleranceConfig( enable_ft_package=True, calc_ft_timeouts=True, simulate_fault=False, simulated_fault_type="random", ) ``` Launch with `ft_launcher` (not `torchrun`): ```bash export GROUP_RANK=0 # required for non-Slurm ft_launcher \ --rdzv_backend=c10d --rdzv_endpoint=${MASTER_ADDR}:${MASTER_PORT} \ --nnodes