ref-hallucination-arena

# Reference Hallucination Arena Skill Evaluate how accurately LLMs recommend real academic references using the OpenJudge `RefArenaPipeline`: 1. **Load queries** — from JSON/JSONL dataset 2. **Collect responses** — BibTeX-formatted references from target models 3. **Extract references** — parse BibTeX entries from model output 4. **Verify references** — cross-check against Crossref / PubMed / arXiv / DBLP 5. **Score & rank** — compute verification rate, per-field accuracy, discipline breakdown 6. **Generate report** — Markdown report + visualization charts ## Prerequisites ```bash # Install OpenJudge pip install py-openjudge # Extra dependency for ref_hallucination_arena (chart generation) pip install matplotlib ``` ## Gather from user before running | Info | Required? | Notes | |------|-----------|-------| | Config YAML path | Yes | Defines endpoints, dataset, verification settings | | Dataset path | Yes | JSON/JSONL file with queries (can be set in config) | | API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. | | CrossRef email | No | Improves API rate limits for verification | | PubMed API key | No | Improves PubMed rate limits | | Output directory | No | Default: `./evaluation_results/ref_hallucination_arena` | | Report language | No | `"en"` (default) or `"zh"` | | Tavily API key | No | Required only if using tool-augmented mode | ## Quick start ### CLI ```bash # Run evaluation with config file python -m cookbooks.ref_hallucination_arena --...