anchor-reviewlisted
Install: claude install-skill sungchul02/anchor-review
# anchor-review
Absolute LLM review scores are unreliable, but rankings are informative. So the manuscript is read blind inside a bundle of papers whose fate is known, by reviewers set to different venue levels.
## Default setup (what a run looks like)
| Item | Default |
|---|---|
| Bundle | **10 papers**: the target plus 9 anchors (3 MAIN, 2 FINDINGS, 2 COLING-accepted, 2 ARXIV-ONLY; 3/3/3 if there are no COLING anchors) |
| Reviewers | **3, one per tier**, each reading **all 10 papers** in its own random order (`presets/tiers.json`) |
| Tiers for target ACL | target: ACL · peer (same level): EMNLP · lower (slightly easier): COLING |
| Scoring | every reviewer scores every paper; a paper's score is the **mean of the 3 reviewers**, and ranking and reading use that mean |
| Output | (1) ranking table: rank, ID, class, mean score, each tier's score, decisive reasons; (2) target's rank against the saved profile; (3) a detailed review of the target |
Change the target venue with `--target-venue` (keys in `presets/tiers.json`; add your own). Change the bundle with `profiles.py sample --mix`.
## Two phases
1. **Build criteria (once per venue × field).**
- Read papers whose outcome is known, *with* their labels, and write one criteria file per reviewer tier. This is phase 1.
2. **Blind review (each manuscript).**
- Mix the manuscript with anchors under hidden names.
- Every tier reviewer reads the whole bundle with its tier's criteria.
- Average the scores and anal