remote-gpu-trainer

Solid

Use when a user runs, debugs, verifies, or ships a DL experiment on a GPU they OWN or RENT (AutoDL, RunPod, vast.ai, Lambda, Paperspace, 恒源云/矩池云/Featurize/揽睿星舟, bare SSH, Slurm, K8s; single/multi-instance). Triggers (multilingual): 本地训练/local training, 远程 GPU 训练/租卡/GPU rental, spot 抢占/preemption, 断点续训/resumable, 防 SSH 断线/tmux 守护, 多实例 ablation, 关机/销毁/stop-vs-terminate billing, checkpoint 磁盘满, CUDA OOM/显存不足, loss NaN/spike/不收敛, overfit 单 batch, FSDP/DeepSpeed/torchrun, 多卡 hang, 训练太慢/GPU util 低, dataloader/数据增广 bug; 消融结果异常/ablation looks wrong, 复现/reproducibility, 数据泄漏/leakage/test-set tuning, mAP=0/全零指标, 输出恒定/model-ignores-input, train-good/val-collapse, 对比不公平/unfair baseline, 单 seed/no error bars, loss 太好/too-good-to-be-true, 跨文档对账/cross-doc drift; 交付产物/deliverable, 唯一真源/single source of truth, best ckpt 拉回, 结果可视化/论文图脚本, manifest/provenance, 一键复现/repro, EVIDENCE.json. NOT for multi-cloud price-shopping + auto spot-recovery (SkyPilot), BYOC dev environments (dstack), or zero-ops serverless inference (Modal).

AI & Automation 63 stars 6 forks Updated 1 months ago MIT

Install

View on GitHub

Quality Score: 85/100

Stars 20%
60
Recency 20%
75
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# remote-gpu-trainer — the DL Experiment Lifecycle ## Overview One skill for the whole arc of a DL experiment: **RUN → VERIFY → DELIVER.** - **RUN** — get a long GPU job to start, survive, and finish, then get the result off the box. On a machine **you own** there is no meter; on a **rented** box the core insight is that **you are a short-term tenant on someone else's machine** — so the job is to *detach the work, make the result outlive the instance, and stop the meter safely*, not to provision a cluster. Platform-specific at the edges (one `profiles/<platform>.md` owns every path, proxy, billing verb, and spot rule), invariant at the core. - **VERIFY** — *is this number a bug, a real effect, or noise?* A surprising result is a hypothesis, not a fact to report. Platform-agnostic. - **DELIVER** — organize the result so every shipped number/figure/table is a *deterministic function of one immutable evidence layer*; provenance and cross-document consistency are locked by mechanism, not by a human remembering to update three documents. Platform-agnostic. Two stances run through VERIFY and DELIVER: **user sovereignty** (the science — seed count, which samples, whether an `aux` channel exists — is the user's call; the skill organizes and discloses a tradeoff *once*, then stops nagging) and **audit → disclose, not enforce** (the skill is an honest auditor, not a gate guard — an integrity issue must surface *with the conclusion it affects*, but the skill never bl...

Details

Author
Hanyuyuan6
Repository
Hanyuyuan6/remote-gpu-trainer
Created
2 months ago
Last Updated
1 months ago
Language
Python
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

remote-gpu-trainer

Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a RENTED or remote instance they do not own — training, eval, ablation sweeps, batch inference, or large data processing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云/矩池云/Featurize/揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual): "远程 GPU 训练", "GPU 租赁", "GPU rental", "租卡", "spot 抢占", "spot preemption", "断点续训", "resumable training", "tmux 训练守护", "防 SSH 断线", "scp/rsync 上传", "多实例 ablation", "远程 GPU 监控", "省钱关机/销毁实例", "stop vs terminate billing", "checkpoint 磁盘满", "CUDA OOM/显存不足", "loss NaN/loss spike", "loss 不下降/不收敛", "overfit 单 batch", "FSDP/DeepSpeed 配置", "多卡训练 hang", "dataloader worker/数据增广 bug". NOT for purely local single-GPU training, in-instance multi-GPU DDP (use torchrun/accelerate), managed multi-cloud price-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).

0 Updated today
Udalefourfold471
AI & Automation Featured

ai-distributed-training

Guides multi-GPU pre-training: DDP, FSDP2, ZeRO, tensor/pipeline/expert parallelism, fp8/Muon. Use when scaling a run, training MoE, or reproducing GPT-2 on rented GPUs.

87 Updated 1 weeks ago
vasilyu1983
AI & Automation Listed

direct

Work autonomously to a finished, verified result instead of narrating options or stopping for permission; spend the cheapest model that cannot get it wrong; report in a short scannable form; and get measurably better each run by recording what actually went wrong. Use when the owner says /direct, "do wonderful things", "finish it", "do it yourself", or otherwise hands over a job and steps away. Encodes evidence discipline, model-tier and token economy, ADHD-friendly reporting, the short list of decisions an agent may never make alone, and a self-evolution loop.

1 Updated 1 weeks ago
rhettcarillo