model-pruning

# Model Pruning: Compressing LLMs ## When to Use This Skill Use Model Pruning when you need to: - **Reduce model size** by 40-60% with <1% accuracy loss - **Accelerate inference** using hardware-friendly sparsity (2-4× speedup) - **Deploy on constrained hardware** (mobile, edge devices) - **Compress without retraining** using one-shot methods - **Enable efficient serving** with reduced memory footprint **Key Techniques**: Wanda (weights × activations), SparseGPT (second-order), structured pruning, N:M sparsity **Papers**: Wanda ICLR 2024 (arXiv 2306.11695), SparseGPT (arXiv 2301.00774) ## Installation ```bash # Wanda implementation git clone https://github.com/locuslab/wanda cd wanda pip install -r requirements.txt # Optional: SparseGPT git clone https://github.com/IST-DASLab/sparsegpt cd sparsegpt pip install -e . # Dependencies pip install torch transformers accelerate ``` ## Quick Start ### Wanda Pruning (One-Shot, No Retraining) **Source**: ICLR 2024 (arXiv 2306.11695) ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer # Load model model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", torch_dtype=torch.float16, device_map="cuda" ) tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf") # Calibration data (small dataset for activation statistics) calib_data = [ "The quick brown fox jumps over the lazy dog.", "Machine learning is transforming the world.", "Artificial int...