moe-training
FeaturedTrain Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.
Install
Quality Score: 94/100
Skill Content
Details
- Author
- Orchestra-Research
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Created
- 1 years ago
- Last Updated
- 1 months ago
- Language
- TeX
- License
- MIT
Integrates with
Similar Skills
Semantically similar based on skill content — not just same category
moe-training
Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.
moe-training
Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.
ml-emerging-techniques
Use when scaling or compressing LLMs with emerging training and inference techniques: Mixture of Experts (MoE) training and expert parallelism (Mixtral, DeepSeekMoE, Switch, DeepSpeed), long-context extension (RoPE, YaRN, ALiBi, position interpolation, FlashAttention), knowledge distillation (soft targets, reverse KLD / MiniLLM, response distillation), model pruning (Wanda, SparseGPT, N:M 2:4 sparsity), model merging with mergekit (linear, SLERP, Task Arithmetic, TIES, DARE, AdaMMS coefficient tuning), and speculative decoding (draft models, Medusa heads, Lookahead decoding, vLLM).