← ClaudeAtlas

anchor-reviewlisted

Blind mock peer review of a manuscript for *ACL venues (ACL, EMNLP, NAACL via ARR). The manuscript is mixed into a 10-paper bundle with same-field "anchor" papers whose outcome is known (main conference, Findings, accepted at a less selective venue, arXiv-only). Three reviewers, one per venue tier (the target venue, a peer venue of the same level, a slightly easier venue), each review the whole bundle. Output is a ranking with scores and reasons, plus a detailed review of the manuscript. Phase 1 builds venue-tier criteria from real papers with known outcomes; phase 2 runs the blind review with them. Users can run the shipped defaults directly or customize venues, paper counts, tiers and bundle size in config. Use when the user asks for a mock review, a blind review, an ARR-style assessment, or "would this get into ACL/EMNLP/NAACL/Findings".
sungchul02/anchor-review · ★ 1 · Code & Development · score 74
Install: claude install-skill sungchul02/anchor-review
# anchor-review Absolute LLM review scores are unreliable, but rankings are informative. So the manuscript is read blind inside a bundle of papers whose fate is known, by reviewers set to different venue levels. ## Default setup (what a run looks like) | Item | Default | |---|---| | Bundle | **10 papers**: the target plus 9 anchors (3 MAIN, 2 FINDINGS, 2 COLING-accepted, 2 ARXIV-ONLY; 3/3/3 if there are no COLING anchors) | | Reviewers | **3, one per tier**, each reading **all 10 papers** in its own random order (`presets/tiers.json`) | | Tiers for target ACL | target: ACL · peer (same level): EMNLP · lower (slightly easier): COLING | | Scoring | every reviewer scores every paper; a paper's score is the **mean of the 3 reviewers**, and ranking and reading use that mean | | Output | (1) ranking table: rank, ID, class, mean score, each tier's score, decisive reasons; (2) target's rank against the saved profile; (3) a detailed review of the target | Change the target venue with `--target-venue` (keys in `presets/tiers.json`; add your own). Change the bundle with `profiles.py sample --mix`. ## Two phases 1. **Build criteria (once per venue × field).** - Read papers whose outcome is known, *with* their labels, and write one criteria file per reviewer tier. This is phase 1. 2. **Blind review (each manuscript).** - Mix the manuscript with anchors under hidden names. - Every tier reviewer reads the whole bundle with its tier's criteria. - Average the scores and anal