using-drac-clusterslisted

# Using DRAC Clusters ## Overview DRAC clusters are shared, MFA-gated, quota-bound Slurm systems. Three properties drive almost every mistake: **you cannot authenticate on your own**, **login nodes are not for computing**, and **almost nothing is knowable without asking the cluster** — accounts, partitions, GPU types and quotas differ per cluster. **Core principle: discover, don't assume.** Every value below has a command that prints the truth. Run it once per cluster and cache it in the project's notes. **Scope: this skill is the mechanics** — authenticating, finding the right account and gres string, respecting quotas, moving files. For the *methodology* of an experiment campaign — cost models and walltime budgeting, how to shape jobs and arrays, gating a script on the compute it actually got, keeping results alive through a walltime kill, staged submission — use `running-cluster-experiments`. ## The MFA gate — read this before anything else **MFA is mandatory on DRAC and an agent cannot complete it.** Non-interactive ssh works *only* by multiplexing over a ControlMaster socket the user already authenticated. Before concluding the cluster is down: ```bash ssh -O check <cluster> # "Master running (pid=...)" = usable ``` With no live master, **every** command fails identically with `Permission denied (keyboard-interactive)`. That is not a transient error, a wrong key, or an outage. Retrying cannot fix it. **Do NOT engineer around the MFA prompt.** It is technica