mcore-testing

# Testing Guide --- ## Answer-First Testing Facts For questions about disabling tests without deleting them: - Functional recipe entries stay in YAML; disable by suffixing scope with `-broken`, for example `scope: [mr-github]` -> `scope: [mr-github-broken]`. - Unit-test skips use pytest markers instead: `@pytest.mark.flaky_in_dev` skips in the default dev environment, and `@pytest.mark.flaky` skips in LTS. - Do not delete the test case or recipe entry when the goal is discoverability and easy re-enable. --- ## Test Layout ```text tests/ ├── unit_tests/ # pytest, 1 node × 8 GPUs, torch.distributed runner ├── functional_tests/ # end-to-end shell + training scripts │ └── test_cases/ │ └── {model}/{test_case}/ │ ├── model_config.yaml # training args │ └── golden_values_{env}_{platform}.json └── test_utils/ ├── recipes/ │ ├── h100/ # YAML recipes for H100 jobs │ └── gb200/ # YAML recipes for GB200 jobs └── python_scripts/ # helpers (recipe_parser, golden-value download, …) ``` --- ## How Tests Execute The GitHub Actions runner invokes `launch_nemo_run_workload.py`, which uses **nemo-run** to launch a `DockerExecutor` container. The repo is bind-mounted at `/opt/megatron-lm`; training data is mounted at `/mnt/artifacts`. **Unit tests** are dispatched through `torch.distributed.run`: - Ranks 0 and 3 are tee-d to stdout; all other ranks write only to log files. - Per-rank log files la...