artifact-detection

# Artifact Detection Tactic Systematically probe benchmarks for annotation artifacts, dataset shortcuts, and spurious correlations that allow models to achieve high scores without the intended capability. ## Stages ### Stage 1: Hypothesis-Only Baseline Test Search literature for evidence that partial-input baselines achieve unexpectedly high performance: - Hypothesis-only baselines (NLI without premise) - Question-only baselines (QA without context) - Label-word frequency baselines - Majority-class and surface-pattern baselines **Search queries**: "[benchmark] annotation artifacts", "[benchmark] hypothesis only", "[benchmark] spurious correlations", "[benchmark] dataset bias" If published partial-input results exist, record performance gap between partial and full input. Gap < 10 points above random indicates severe artifacts. ### Stage 2: Contrast Set Construction Identify whether contrast sets or adversarial evaluations exist: - Search for "[benchmark] contrast sets", "[benchmark] adversarial examples" - Check if CheckList-style behavioral tests have been applied - Look for counterfactual data augmentation studies Record performance drops on contrast sets. Drops > 20 points indicate reliance on surface patterns. ### Stage 3: Format Manipulation Probes Search for evidence of format sensitivity: - Prompt template sensitivity studies - Label name/ordering effects - Verbalization effects in classification - Input length correlations with labels Record whether minor ...