huggingface-accelerate

# HuggingFace Accelerate - Unified Distributed Training ## Quick start Accelerate simplifies distributed training to 4 lines of code. **Installation**: ```bash pip install accelerate ``` **Convert PyTorch script** (4 lines): ```python import torch + from accelerate import Accelerator + accelerator = Accelerator() model = torch.nn.Transformer() optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset) + model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: optimizer.zero_grad() loss = model(batch) - loss.backward() + accelerator.backward(loss) optimizer.step() ``` **Run** (single command): ```bash accelerate launch train.py ``` ## Common workflows ### Workflow 1: From single GPU to multi-GPU **Original script**: ```python # train.py import torch model = torch.nn.Linear(10, 2).to('cuda') optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) for epoch in range(10): for batch in dataloader: batch = batch.to('cuda') optimizer.zero_grad() loss = model(batch).mean() loss.backward() optimizer.step() ``` **With Accelerate** (4 lines added): ```python # train.py import torch from accelerate import Accelerator # +1 accelerator = Accelerator() # +2 model = torch.nn.Linear(10, 2) optimizer = torch.optim.Adam(model.parameters()) dataloader = to...