padosoft-ai-evaluationlisted

# AI evaluation harness A model-backed feature has no failing test to point at: the output is different every time and often plausibly wrong. So the question "did this change make it better?" is only answerable with a **measurement** — and a measurement that cannot be reproduced or compared is a number somebody wrote down. **The rule: a number without a dataset version and a run identity is not a measurement.** --- ## 1. The dataset and the report are versioned artifacts, not files Both carry a **schema version**, so a consumer can reject what it does not understand instead of misreading it. - **Absent version means the original version**, for backward compatibility. An explicit unsupported one **fails at load**, loudly. - **The value objects validate the version in their constructor**, not only the loader. Consumers construct them directly, and a check that lives in the parser protects nobody who did not go through the parser. - **Reject mixed sources at the boundary.** If a dataset can come from a file or be built in code, taking metadata from one and samples from the other produces a report about something that never existed. Replacing a file-backed set with another file-backed set is fine; switching kind is not. - **Report identity**: the dataset version, the system version, the model and its parameters, the prompt revision, the timestamp. Two reports are comparable only if those match, and the harness is what should say so rather than the person reading