coreweave-data-handling

# CoreWeave Data Handling ## Overview CoreWeave GPU cloud workloads involve large-scale data artifacts: model weights (multi-GB safetensors/GGUF), training datasets (parquet, TFRecord, WebDataset), checkpoint snapshots, and inference cache volumes. Data flows through Kubernetes PersistentVolumeClaims backed by region-specific storage classes. Compliance requires encryption at rest via the storage driver, namespace-scoped RBAC for volume access, and audit logging for any data egress from GPU nodes. ## Data Classification | Data Type | Sensitivity | Retention | Encryption | |-----------|-------------|-----------|------------| | Model weights | Medium | Until deprecated | AES-256 at rest | | Training datasets | High (may contain PII) | Per data license | AES-256 + TLS in transit | | Checkpoint snapshots | Medium | 30 days post-training | AES-256 at rest | | Inference cache | Low | Session/TTL | Volume-level encryption | | HuggingFace tokens | Critical | Rotate quarterly | K8s Secret + KMS | ## Data Import ```typescript import { KubeConfig, BatchV1Api } from '@kubernetes/client-node'; async function importDataset(pvcName: string, sourceUrl: string, namespace: string) { const kc = new KubeConfig(); kc.loadFromDefault(); const batch = kc.makeApiClient(BatchV1Api); const job = { metadata: { name: `import-${Date.now()}`, namespace }, spec: { template: { spec: { restartPolicy: 'Never', containers: [{ name: 'loader', image: 'python:3.11-slim', ...