llm-eval-harness-and-scoring-pipelinelisted

# LLM eval harness and scoring pipeline Any pipeline that turns LLM output into a number or a structured decision (a grade, a rank, a composite score) needs three things most first drafts skip: locked-down math that can't silently drift, an explicit policy for what happens when part of the pipeline fails, and a way to measure quality changes before they ship. This skill covers all three, plus the shadow-comparison technique for testing changes safely. ## Lock the aggregation math, and gate changes on it explicitly Whatever formula turns multiple sub-scores into one number (a mean, a weighted composite, a rank with a tie-breaking rule) should be written down as an explicit, versioned contract: not just "whatever the code currently does." Once real scores exist that people compare over time, changing this math retroactively changes the meaning of every past score. Treat any change to it as requiring the same sign-off as a database migration: explicit, documented, and updated in one commit alongside whichever doc is the source of truth for it. ## Partial failure: degrade, never silently substitute The most expensive bug class in a multi-call scoring pipeline is emitting a comparable-looking number that quietly lost an input. Concretely: if one sub-call in a multi-call scoring flow fails, - **If the failed piece is not load-bearing for the final number** (a non-critical axis, an optional embellishment), renormalize over what succeeded and keep the result flagged as complete