sglang

# SGLang High-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching. ## When to use SGLang **Use SGLang when:** - Need structured outputs (JSON, regex, grammar) - Building agents with repeated prefixes (system prompts, tools) - Agentic workflows with function calling - Multi-turn conversations with shared context - Need faster JSON decoding (3× vs standard) **Use vLLM instead when:** - Simple text generation without structure - Don't need prefix caching - Want mature, widely-tested production system **Use TensorRT-LLM instead when:** - Maximum single-request latency (no batching needed) - NVIDIA-only deployment - Need FP8/INT4 quantization on H100 ## Quick start ### Installation ```bash # pip install (recommended) pip install "sglang[all]" # With FlashInfer (faster, CUDA 11.8/12.1) pip install sglang[all] flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/ # From source git clone https://github.com/sgl-project/sglang.git cd sglang pip install -e "python[all]" ``` ### Launch server ```bash # Basic server (Llama 3-8B) python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --port 30000 # With RadixAttention (automatic prefix caching) python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --port 30000 \ --enable-radix-cache # Default: enabled # Multi-GPU (tensor parallelism) python -m sglang.launch_server \ --model-path meta-llama/Meta-Ll...