optimizing-pyspark-jobslisted

# Optimizing PySpark Jobs ## When to use - A Spark/PySpark job is slow, spills to disk, or OOMs (driver or executor). - One or a few tasks straggle while the rest finish (skew). - Huge shuffles, wide stages, or exploding output. - Do NOT use for pure SQL warehouse tuning (use `optimizing-sql-queries`). ## Workflow ``` - [ ] Read the Spark UI: find the slow stage and its shuffle/skew - [ ] Confirm the cause: skew, too many/few partitions, wide shuffle, or driver pull - [ ] Fix joins (broadcast small side; salt skewed keys) - [ ] Right-size partitions; enable AQE - [ ] Re-run and compare stage time/shuffle bytes ``` 1. **Read the Spark UI** (Stages/SQL tab). Find the stage dominating wall-clock; look at shuffle read/write and the task-duration distribution (a long tail = skew). 2. **Diagnose** the dominant cause before changing config. 3. **Fix joins first** — broadcast the small side; handle skewed keys. 4. **Right-size partitions** and let Adaptive Query Execution coalesce them. 5. **Re-measure** in the UI; confirm shuffle bytes / stage time dropped. ## Patterns **Broadcast the small side** to avoid a shuffle join: ```python from pyspark.sql.functions import broadcast fact.join(broadcast(small_dim), "dim_id") ``` **Enable Adaptive Query Execution** (coalesces partitions, converts to broadcast, handles skew automatically): ```python spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") ``` **Salt a s