optimize-promptslisted

# Optimize prompts (with a method, not vibes) Hand-tuning a prompt and eyeballing one output tops out fast and quietly overfits to the last example you looked at. Systematic prompt optimization treats the prompt as something you search over, scored by an eval set. ## Prerequisite: you need an eval set You cannot optimize what you cannot measure. Build an eval set first (`build-eval-dataset`) and wire up scoring (`add-llm-evals`). The optimization loop is: propose a prompt variant, score it on the eval set, keep the winner. Everything below is a smarter way to propose variants. ## The levers, cheapest first 1. **Instructions.** Clarify the task, add constraints, specify the output format. The highest-leverage and cheapest change. 2. **Few-shot examples.** Adding 2-5 good examples often beats a longer instruction. *Which* examples matters a lot; select them from your eval/production data, and measure (more examples is not always better, and they cost input tokens). 3. **Output structure.** Ask for structured output (JSON/enum) when you need reliability; add a short reasoning step before the answer when quality needs it. 4. **Decomposition.** Split one overloaded prompt into a small pipeline of focused steps (each independently evaluable). ## Automate the search (when hand-tuning plateaus) Once you have an eval metric, use an optimizer instead of manual trial-and-error: - **Auto instruction search (APE-style):** have an LLM propose many instruction candidates, score each