design-resumable-model-evaluationlisted

# Design Resumable Model Evaluation Make interruption cheap and evidence comparable. ## Freeze an evaluation revision Create a unique revision for every combination of corpus membership, model, prompt/instructions, schema, decoder settings, validator, run count, and thresholds. Hash this configuration into the state file. Never seed a new revision from incompatible results. Write the case manifest before inference. Select cases from stable metadata, not previous outcomes. ## Persist atomically Use an append-only result stream or atomically replace a small state file after every case/run. Store: - case and run IDs; - frozen configuration hash; - raw and normalized output; - parse, identifier, semantic, execution, and safety gate results; - abstention and generation-failure status; - start/end timestamps and latency; - interruption and retry metadata. On resume, validate the configuration hash, scan completed keys, and execute only missing `(case, run)` pairs. Deduplicate safely if a crash occurred after writing output but before updating summary state. ## Preserve benchmark semantics Evaluate first-attempt output with exactly one generation when that is the product question. Do not call production remediation, repair malformed JSON, or deterministically choose a fallback. Exercise remediation in a separate test and metric series. Keep required run counts and thresholds unchanged. A resumable evaluator improves execution efficiency, not acceptance criteria. ## Stop