video-model-bakeofflisted
Install: claude install-skill thepandanlabs/video-gen-demo-booth
# Video model bake-off
Pick a provider and a model with evidence from your own content, not from a leaderboard.
## The core claim
**Leaderboards measure preference, not your job.** The highest-ranked image-to-video model
on Artificial Analysis will lose your bake-off if your job is "keep this person's face" and
the model quietly swaps in someone else's. That has happened: a model ranked #1 for
image-to-video produced a visibly different person by the final frame in every cell, while
scoring fine on general aesthetics.
Run the bake-off. It costs about $10 and takes an hour.
## Flow
```
1. Frame the job what must be true of the output, in one sentence
2. Survey providers who hosts models, at what price, on what terms
3. Survey models shortlist 4-6 arms, including the incumbent
4. Build the assets test subjects and scenes (IP-safe if archetypal)
5. Run the harness every arm x subject x plate, checkpointed
6. Score by frame pairs first frame vs final frame, not by watching
7. Publish the comparison a page with clips, scores, cost and latency
```
Do not skip 1. Everything else is downstream of it.
## 1. Frame the job
Write one sentence: *what must be true of the output for this to work at all?*
Examples that produce different winners:
- "The attendee must recognise themselves" → identity preservation is the gate
- "The product must look exactly like the SKU" → object fidelity is the gate
- "It must feel cinematic" → aesthetics is th