spark-optimizerlisted
Install: claude install-skill satyaprakashdwivedi/spark-optimizer-skill
# Spark optimizer
Turn Spark logs into a measured diagnosis and a concrete, conflict-free config. The bundled analyzer does the
measuring. Your job is to check its findings against the user's context and to explain the trade-offs.
## 1. Collect inputs
Ask only for what's missing. One message with all the questions is fine.
- **Logs (required).** Any of:
- **Spark event log:** the most useful input. Get it from the History Server → app → *Download*, or from the
`spark.eventLog.dir` path, e.g. `hdfs:///spark-logs/application_…`, an S3 or DBFS path, or an `eventlog_v2_*`
folder for rolling logs.
- **Driver or executor stderr:** from `yarn logs -applicationId <id>`, `kubectl logs`, or the Databricks/EMR log
folders.
- A folder that contains either kind.
- `.gz` files are read directly. Files compressed with lz4, zstd or snappy need decompressing first
(`zstd -d file.zstd`, `lz4 -d file.lz4`).
- **Cluster shape (strongly recommended):** cores and memory per worker node, and the number of nodes. Without
them the analyzer can only adjust executors relative to the current settings.
- **Context:** the SLA or goal (fastest, cheapest, or just stop failing), whether the cluster is shared, and the
platform (YARN, K8s, Databricks, EMR).
## 2. Run the analyzer
```bash
python3 <skill-dir>/scripts/analyze_spark_log.py <logs...> \
[--node-cores 16 --node-mem-gb 64 --nodes 6] [--target-partition-mb 128] [--json] [--out report.md]
```
- `<skill-dir>` is the