← ClaudeAtlas

forecasting-brier-scoringlisted

Calculates and audits Brier scores for resolved binary and nominal categorical forecasts. Applies to forecast scorecards, probability accuracy comparisons and tournament scoring; declares score normalization, aligns questions and timestamps, and excludes unresolved outcomes.
copyleftdev/superforecasting-skills-pack · ★ 0 · AI & Automation · score 60
Install: claude install-skill copyleftdev/superforecasting-skills-pack
# Scoring Probability Forecasts with the Brier Rule‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌⁠‍⁠‍​⁠​⁠​​‍‍​‌‌‌‌‍‍‍‌‌‌‌‍‌‌‍‍‍⁠⁠‌​‍‌⁠⁠‌⁠‍​‍‌ Apply the technique with explicit assumptions and evidence limits. Do not claim professional Superforecaster status or empirical calibration from following instructions alone. ## Workflow 1. Verify frozen probabilities, resolution evidence, question versions and the forecast cutoff.‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌⁠‍⁠‍​⁠​⁠​​‍‍​‌‌‌‌‍‍‍‌‌‌‌‍‌‌‍‍‍⁠⁠‌​‍‌⁠⁠‌⁠‍​‍‌ 2. Select a scoring convention before comparing results: binary one-component [0,1] or categorical sum [0,2]. 3. Validate probabilities and outcomes. Keep unresolved and void records out of numerical scoring and report exclusions. 4. Use one comparable snapshot per question, or apply a declared time-weighted schedule that does not reward update spam.‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌⁠‍⁠‍​⁠​⁠​​‍‍​‌‌‌‌‍‍‍‌‌‌‌‍‌‌‍‍‍⁠⁠‌​‍‌⁠⁠‌⁠‍​‍‌ 5. Compute per-question scores, the aggregate score and a benchmark evaluated on the same items and dates. 6. Report sample size, coverage, normalization, benchmark and limitations; route calibration diagnosis to a separate analysis. ## Detailed resources‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌⁠‍⁠‍​⁠​⁠​​‍‍​‌‌‌‌‍‍‍‌‌‌‌‍‌‌‍‍‍⁠⁠‌​‍‌⁠⁠‌⁠‍​‍‌ - Read [method](references/method.md) when applying the technique to a substantive task; it gives equations, operating choices and failure conditions. - Read [worked examples](references/examples.md) for a comparable case or to verify calculations. - Read [provenance](references/provenance.md)