ml-multimodal-roboticslisted
Install: claude install-skill Lu1sDV/skillsmd
# Multimodal ML & Robotics VLA
Three domain references, one per workflow family. Open only the reference the
task needs; do not preload all.
| Task / Tool | Read |
|---|---|
| CLIP, BLIP-2, LLaVA, Segment Anything (SAM) — zero-shot classification, captioning, VQA, visual chat, promptable/auto segmentation | [Vision-language & segmentation](references/vision-language.md) |
| Stable Diffusion (Diffusers), Whisper, AudioCraft (MusicGen, AudioGen, EnCodec) — image generation, LoRA/ControlNet, ASR, audio/music generation | [Image generation & audio](references/generation-audio.md) |
| OpenPI (`pi0`, `pi0-fast`, `pi0.5`), OpenVLA-OFT / OFT+, NVIDIA Cosmos Policy — VLA fine-tuning, norm stats, WebSocket/HTTP serving, LIBERO/ALOHA/DROID/RoboCasa eval | [Robotics VLA policies](references/robotics-vla.md) |
## What each covers
- **Vision-language & segmentation** — tool selection across CLIP, BLIP-2,
LLaVA, and SAM; model variants and FP16/INT8/INT4 VRAM budgets; point/box/mask
prompting and ONNX export for SAM.
- **Image generation & audio** — Diffusers pipelines, schedulers, LoRA
blending, ControlNet, and VRAM offloading; Whisper model sizing, CLI/Python
transcription, and `faster-whisper`; AudioCraft MusicGen (text/melody/style),
AudioGen, and EnCodec sample-rate rules.
- **Robotics VLA policies** — OpenPI JAX/PyTorch fine-tuning, `compute_norm_stats.py`,
`transformers_replace` patching, and WebSocket serving; OpenVLA-OFT/OFT+
pinned dependencies, LoRA merging, and