experiment-design

# Experiment Design Methodology You are helping a researcher design rigorous experiments. Follow this methodology systematically. ## Step 1: Understand the Research Question Before designing any experiment: - Ask what specific hypothesis or claim the experiment should support - Identify the dependent variable (metric) and independent variables (factors) - Clarify the baseline: what is the current best result or default configuration? ## Step 2: Single-Variable Isolation Every ablation study must change exactly ONE variable at a time. For each factor: 1. **Define the factor** — what is being varied (e.g., loss function, learning rate, architecture component) 2. **List levels** — all values this factor will take (e.g., CE, focal, VAR) 3. **Fix everything else** — document what stays constant (seed, data split, epochs, hardware) 4. **Predict outcome** — before running, state what you expect and why Template for each ablation row: ``` | Run ID | Factor | Value | Fixed Config | Expected Outcome | |--------|--------|-------|-------------|-----------------| ``` ## Step 3: Experiment Matrix For multi-factor studies, use a structured matrix: 1. **Full factorial** — if factors are few (≤3) and levels are few (≤3 each) 2. **Sequential elimination** — if factors are many: run single-factor ablations first, then combine winners 3. **Latin square** — if full factorial is too expensive: sample representative combinations Always calculate total runs before committing: ``` Total ru...