model-selection

# Model Selection ## Purpose Choose the model that meets the task's requirements at the lowest cost and latency. Most production LLM features run on a model several times more expensive than the task requires, because nobody measured the cheaper one. ## When to Use - Choosing a model for a new feature. - Reducing the cost or latency of an existing feature. - Evaluating whether a newly released model is worth migrating to. - Designing a routing strategy across several models. ## Capabilities - Task-to-capability matching. - Cost and latency modeling at real volume. - Routing: cheap model first, escalate on difficulty. - Benchmark interpretation and its limits. - Migration and re-evaluation. ## Inputs - The task, and the accuracy it genuinely requires. - Volume, latency budget, and cost budget. - An evaluation set — without one, this is guesswork. ## Outputs - A model choice justified by measurement on your data. - A cost and latency projection at real volume. - A routing strategy, where one is warranted. ## Workflow 1. **Start with the cheapest plausible model** — Not the best one. Measure it on your evaluation set. Escalate only if it fails, and only as far as necessary. 2. **Measure on your data, not on benchmarks** — A model that leads on MMLU may be worse at your specific extraction task. Public benchmarks measure general capability, and your task is not general. 3. **Model the cost at real volume** — A per-call cost difference that looks trivial becomes the do...