spark-scala-guidelineslisted

# Spark Scala Guidelines Language-specific Spark 4 rubric for Scala agents. Always apply `spark-guidelines` first for engine-level pipeline design; use this skill for Scala idioms. General Scala style still applies — immutability-first, exhaustive matching over sealed ADTs, no `null` or `Await.result` in domain code; for substantial non-Spark Scala modules, defer to the scala-implementer or scala-reviewer agents, which carry the full `scala-guidelines` rubric. ## Source Of Truth - Spark Scala API docs: https://spark.apache.org/docs/latest/api/scala/ - Structured Streaming `transformWithState` Scala examples in the official guide - The repo's `build.sbt`/`build.mill` — Spark version, Scala version (2.12 vs 2.13 vs 3.x), and shading rules ## API Choice - **DataFrame by default** for ETL breadth and SQL interoperability. Reach for **Dataset[T]** when: - The element type is stable and encoded with a case class + `Encoders.product` - You need compile-time type safety on map/filter operations that would be error-prone as string column names - You are building a reusable library API consumed by other Scala modules - Do not use Dataset for pipelines that are primarily SQL or PySpark-portable — the encoding overhead and version coupling rarely pay off. - **Never use RDD** in new Spark 4 pipeline code unless interfacing with legacy libraries that have no DataFrame equivalent. ## Idiomatic Transforms - Express transforms as **Column expressions** and `select`/`withColumn` c