compare

Solid

Same-epoch comparison and analysis of training runs across wandb, neptune, tensorboard, mlflow, or local result files. Aligns runs at the student's current step (never current-vs-final-of-baseline) and separates proxy metrics from downstream targets. Also writes run summaries and experiment reports, analyzes ablations, and monitors training. Use when the user asks to compare runs, check if a run is improving, track lag against a baseline, rank experiments, evaluate run-vs-run performance, analyze results, write a training report or experiment summary, summarize training results, monitor training, or asks which run is best.

AI & Automation 461 stars 38 forks Updated 3 days ago MIT

Install

View on GitHub

Quality Score: 91/100

Stars 20%
89
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Compare: same-epoch run comparison across trackers The most common comparison error is reporting "run A is 4 percentage points behind baseline" when run A is at epoch 11 of 100 and the baseline number is from epoch 100. The student is _still training_; the comparison is meaningless. This skill enforces same-epoch alignment. The agentic Stop hook routes here from `reason` when an assistant reports a delta without aligning the runs. If the question is "why did X fail / diverge / underperform", go to `/labrat:debug` first. ## Auto-detect the tracker Check in this order: 1. `WANDB_API_KEY` env var set, or `wandb` imports or a `*.yaml` config with wandb keys in the project → **wandb** 2. `NEPTUNE_API_TOKEN` env var set, or a `*.yaml` config with neptune keys → **neptune** 3. `MLFLOW_TRACKING_URI` env var set, or `mlruns/` dir present → **mlflow** 4. `runs/` or `lightning_logs/` dir present → **tensorboard** 5. `*results*.json` / `*meta*.json` / `eval*.csv` files in run dirs → **local file format**: read one sample file and write the analysis code for that format, not a hardcoded script If none, ask the user where metrics live before guessing. ## The protocol ### 1. Identify the runs Get full names (no shortcodes). If the user says "run-b vs the baseline", clarify: - which `run-b` run (project + entity + run-id) - which baseline (full run name; baselines often have several variants) ### 2. Fetch metric history (not just final value) You need the full curve, not the l...

Details

Author
fcakyon
Repository
fcakyon/labrat
Created
8 months ago
Last Updated
3 days ago
Language
Python
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category