llm-calibration-logprobslisted

# Reading a Model's Own Uncertainty from Token Log-Probabilities ## Instructions This skill covers **within-model** confidence: how sure a single model is about each decision it makes, read off the token log-probabilities it emits, and whether that internal confidence is *calibrated* against ground truth. It pairs with the `model-council-voting` skill, which handles the complement — **between-coder** agreement across several independent models. Use both: one model's high self-reported confidence on an item, and three models independently agreeing on that item, are different kinds of evidence, and a careful pipeline reports both. This skill does not cover codebook design or human-validation statistics (κ, F1) — those live in the `text-classification` skill; cross-reference it rather than re-deriving them here. ### 1. What a Token Log-Probability Is, and When to Use It - A generative LLM produces each output token by sampling from a probability distribution over its vocabulary. The **log-probability** (logprob) of the token it actually emitted is `log P(token | context)`; exponentiating gives a probability in `[0, 1]`. A logprob of `0.0` means probability 1.0 (the model treated that token as certain); `-2.30` means probability ≈ 0.10. This is the model's *own* assessment of how likely that token was, conditional on everything before it. - Use logprobs when you need a **per-item, per-decision** confidence signal that is nearly free (it falls out of the same forward pass that