dask

# Dask ## Overview Dask is a Python library for parallel and distributed computing that enables three critical capabilities: - **Larger-than-memory execution** on single machines for data exceeding available RAM - **Parallel processing** for improved computational speed across multiple cores - **Distributed computation** supporting terabyte-scale datasets across multiple machines Dask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs. ## When to Use This Skill This skill should be used when: - Process datasets that exceed available RAM - Scale pandas or NumPy operations to larger datasets - Parallelize computations for performance improvements - Process multiple files efficiently (CSVs, Parquet, JSON, text logs) - Build custom parallel workflows with task dependencies - Distribute workloads across multiple cores or machines ## Core Capabilities Dask provides five main components, each suited to different use cases: ### 1. DataFrames - Parallel Pandas Operations **Purpose**: Scale pandas operations to larger datasets through parallel processing. **When to Use**: - Tabular data exceeds available RAM - Need to process multiple CSV/Parquet files together - Pandas operations are slow and need parallelization - Scaling from pandas prototype to production **Reference Documentation**: For comprehensive guidance on Dask DataFrames, refer to `references/dataframes.md` which includes: - Reading data (single file...