llm-campaign-drift-gate

# LLM Campaign Drift Gate ## Problem Batch-scoring campaigns (exposure measures, classifiers, extraction runs) call provider aliases that can be silently repointed to a new model at any time. Resuming a half-finished campaign after the alias moves splices two different scorers into one variable, with the version boundary correlated with whatever orders the chunks (time, firm id) — a silent confound. Providers can also RETIRE the old model entirely, making the original campaign uncompletable. ## Context / Trigger Conditions - Resuming a scoring run more than ~a day after its last paid chunk - "Top up credits and finish the run" requests - Any incremental scoring against an existing response cache - Symptom of a missed gate: a step-change in scores at a resume boundary ## Solution Before ANY production spend on resume, run a two-part gate (~$0.30–2): 1. **Canary (the decisive check):** sample ~100 already-cached items, re-send their EXACT stored prompts fresh, compare fresh vs cached scores. Gate: ≥97% all-field exact match and no systematic directional shift. Write the comparison in a standalone script — never through the pipeline's cache layer, which would overwrite production entries. 2. **Gold re-validation:** re-score the gold/validation panel fresh and compare agreement metrics to the prior validation (e.g. median F1/κ within ~0.03, no domain dropping >0.10). Also capture `response.model` on every gate call — pipelines rarely store it, and it is ...