author-calibrated-evallisted

# Author-Calibrated Eval The ground truth for whether generated prose is "worth reading" lives in the eyes of the person who reads it (the author). Build the loop on that premise. Split the roles three ways. | What is checked | Who checks it | |---|---| | Readability and reading value (taste) | **The author**'s blind side-by-side read | | Faithfulness (things not in the materials, comparison target, qualifiers, substituted subject) | **LLM judge**. A cutoff of binary checks | | Format (position of citation marks, presence of markers, character count) | **Code** | Asking the LLM judge for a holistic readability verdict does not make it a substitute for the author. Measured (2026-09-23, jev-research-pipeline): even with a rubric that stated explicitly "having more information is not a reason to win", the Opus judge picked the more information-dense version as "easier to understand" in both orderings, while the author called the short version "very easy to understand" and the information-dense version "hard to read". On the other hand, the judge caught, in both orderings, a substituted comparison target that the author had not noticed while reading. ## 1. Fix the materials - Fix, one case at a time, the input used when the actual output was produced (that day's full set of materials). If the materials include third-party original text, store them in a private location, not in a public repo - Assign each case to dev or holdout. Derive the assignment from the case id alone