forecasting-ch09-expert-scoringlisted

# Chapter 9: The Oracle Problem ## Scope and intake Use when probability forecasters need fair scoring and calibration diagnostics on common resolved events. Do not use for ranking experts by reputation, one success or noncomparable question sets. Ask only for unresolved material inputs: Which events and forecast lead time are shared? Were forecasts recorded before resolution? What baseline was predeclared? Are outcomes missing or selectively reported? ## Input contract and additional evidence CSV event_id,probability,outcome,baseline; probabilities and baseline in [0,1], outcome 0/1 for CLI scoring; retain unresolved cases in the journal, not this scoring file. Each row is one eligible event forecast. Multiple experts require separate aligned forecast columns/files or an explicitly adapted expert_id panel; revisions must be reduced by the predeclared cutoff rule. Minimal **format illustration**, not sufficient training data: ```csv event_id,probability,outcome,baseline E1,0.7,1,0.5 E2,0.2,0,0.5 ``` ## Executable interface Exact CLI columns: `event_id,probability,outcome,baseline`. Supported method controls: `as_of, frequency, horizon, season, seed`. The scoring adapter requires one resolved outcome per event and uses fixed probability bins. Wilson frequency bounds assume independent events. Preserve unresolved/late forecasts outside this scoring input and report their exclusions in the accompanying analysis. ## Applied procedure 1. Fix event eligibility and lead