evaluating-llms-harness

# lm-evaluation-harness - LLM Benchmarking ## Quick start lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. **Installation**: ```bash pip install lm-eval ``` **Evaluate any HuggingFace model**: ```bash lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag \ --device cuda:0 \ --batch_size 8 ``` **View available tasks**: ```bash lm_eval --tasks list ``` ## Common workflows ### Workflow 1: Standard benchmark evaluation Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). Copy this checklist: ``` Benchmark Evaluation: - [ ] Step 1: Choose benchmark suite - [ ] Step 2: Configure model - [ ] Step 3: Run evaluation - [ ] Step 4: Analyze results ``` **Step 1: Choose benchmark suite** **Core reasoning benchmarks**: - **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice - **GSM8K** - Grade school math word problems - **HellaSwag** - Common sense reasoning - **TruthfulQA** - Truthfulness and factuality - **ARC** (AI2 Reasoning Challenge) - Science questions **Code benchmarks**: - **HumanEval** - Python code generation (164 problems) - **MBPP** (Mostly Basic Python Problems) - Python coding **Standard suite** (recommended for model releases): ```bash --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge ``` **Step 2: Configure model** **HuggingFace model**: ```bash lm_eval --model hf \ --model_args pretrained=meta-llama/Ll...