nanogpt

# nanoGPT - Minimalist GPT Training ## Quick start nanoGPT is a simplified GPT implementation designed for learning and experimentation. **Installation**: ```bash pip install torch numpy transformers datasets tiktoken wandb tqdm ``` **Train on Shakespeare** (CPU-friendly): ```bash # Prepare data python data/shakespeare_char/prepare.py # Train (5 minutes on CPU) python train.py config/train_shakespeare_char.py # Generate text python sample.py --out_dir=out-shakespeare-char ``` **Output**: ``` ROMEO: What say'st thou? Shall I speak, and be a man? JULIET: I am afeard, and yet I'll speak; for thou art One that hath been a man, and yet I know not What thou art. ``` ## Common workflows ### Workflow 1: Character-level Shakespeare **Complete training pipeline**: ```bash # Step 1: Prepare data (creates train.bin, val.bin) python data/shakespeare_char/prepare.py # Step 2: Train small model python train.py config/train_shakespeare_char.py # Step 3: Generate text python sample.py --out_dir=out-shakespeare-char ``` **Config** (`config/train_shakespeare_char.py`): ```python # Model config n_layer = 6 # 6 transformer layers n_head = 6 # 6 attention heads n_embd = 384 # 384-dim embeddings block_size = 256 # 256 char context # Training config batch_size = 64 learning_rate = 1e-3 max_iters = 5000 eval_interval = 500 # Hardware device = 'cpu' # Or 'cuda' compile = False # Set True for PyTorch 2.0 ``` **Training time**: ~5 minutes (CPU), ~1 minute...