cupynumeric-parallel-data-load

# Parallel sharded data -> cupynumeric load **Why this skill exists.** cupynumeric mirrors NumPy's array API, including `cupynumeric.load` for a single `.npy` file. Beyond that, file *loading* lives in Legate, not cupynumeric: | Format | Built-in loader | |---|---| | Single `.npy` | `cupynumeric.load(path)` (NumPy-API parity) | | HDF5 (single file) | `legate.io.hdf5.from_file` / `from_file_batched` | | Sharded multi-file (any format), Parquet/Arrow, raw binary, custom layouts | **No built-in loader — this skill.** | This skill shows the canonical way to fill the gap in the last row: write a Legate Python task that calls the third-party reader the format needs (`h5py`, `pyarrow`, `np.memmap`, ...) inside the task body, and let Legate distribute the reads across GPUs / nodes. For the formats with a built-in loader, prefer it unless you need a custom in-task body (mmap-based loader, format-specific decoder, sidecar metadata, partial / sharded reads). Canonical pattern: **manual partition + manual task launch, sized to the machine, not the files.** Only axis 0 is sharded; trailing axes ride along inside each tile. Per-shard row counts may differ across files (only `dtype` and trailing axes must match); the launch fills every available processor regardless of how many files there are. `.npy` is the worked example because the header carries shape and dtype on disk, but the skeleton applies to any format with cheap range/slice reads (raw binary, HDF5, Parquet/Arrow — see "Other...