← ClaudeAtlas

ml-multimodal-roboticslisted

Use when working with multimodal ML models or vision-language-action (VLA) robotics policies: zero-shot image-text matching and retrieval (CLIP), image captioning and VQA (BLIP-2), conversational visual chat (LLaVA), promptable segmentation (Segment Anything / SAM), text-to-image diffusion and LoRA/ControlNet (Stable Diffusion / Diffusers), speech transcription and translation (Whisper), music and sound-effect generation (AudioCraft / MusicGen / AudioGen), or robot policy fine-tuning, serving, and simulation eval (OpenPI pi0/pi0.5, OpenVLA-OFT, NVIDIA Cosmos Policy on LIBERO, ALOHA, DROID, RoboCasa).
Lu1sDV/skillsmd · ★ 2 · AI & Automation · score 63
Install: claude install-skill Lu1sDV/skillsmd
# Multimodal ML & Robotics VLA Three domain references, one per workflow family. Open only the reference the task needs; do not preload all. | Task / Tool | Read | |---|---| | CLIP, BLIP-2, LLaVA, Segment Anything (SAM) — zero-shot classification, captioning, VQA, visual chat, promptable/auto segmentation | [Vision-language & segmentation](references/vision-language.md) | | Stable Diffusion (Diffusers), Whisper, AudioCraft (MusicGen, AudioGen, EnCodec) — image generation, LoRA/ControlNet, ASR, audio/music generation | [Image generation & audio](references/generation-audio.md) | | OpenPI (`pi0`, `pi0-fast`, `pi0.5`), OpenVLA-OFT / OFT+, NVIDIA Cosmos Policy — VLA fine-tuning, norm stats, WebSocket/HTTP serving, LIBERO/ALOHA/DROID/RoboCasa eval | [Robotics VLA policies](references/robotics-vla.md) | ## What each covers - **Vision-language & segmentation** — tool selection across CLIP, BLIP-2, LLaVA, and SAM; model variants and FP16/INT8/INT4 VRAM budgets; point/box/mask prompting and ONNX export for SAM. - **Image generation & audio** — Diffusers pipelines, schedulers, LoRA blending, ControlNet, and VRAM offloading; Whisper model sizing, CLI/Python transcription, and `faster-whisper`; AudioCraft MusicGen (text/melody/style), AudioGen, and EnCodec sample-rate rules. - **Robotics VLA policies** — OpenPI JAX/PyTorch fine-tuning, `compute_norm_stats.py`, `transformers_replace` patching, and WebSocket serving; OpenVLA-OFT/OFT+ pinned dependencies, LoRA merging, and