← ClaudeAtlas

spark-optimizerlisted

Analyse Apache Spark logs and recommend the best Spark configuration. Use when the user shares or points to a Spark event log, History Server download, eventlog_v2 folder, or driver/executor stderr (YARN, Kubernetes, EMR, Databricks, Dataproc). Also use when they ask to tune or speed up a Spark job, or to fix OOM, "Container killed by YARN", spill, skew, slow stages, GC or shuffle fetch failures, or to size executors, memory, cores and spark.sql.shuffle.partitions.
satyaprakashdwivedi/spark-optimizer-skill · ★ 0 · DevOps & Infrastructure · score 72
Install: claude install-skill satyaprakashdwivedi/spark-optimizer-skill
# Spark optimizer Turn Spark logs into a measured diagnosis and a concrete, conflict-free config. The bundled analyzer does the measuring. Your job is to check its findings against the user's context and to explain the trade-offs. ## 1. Collect inputs Ask only for what's missing. One message with all the questions is fine. - **Logs (required).** Any of: - **Spark event log:** the most useful input. Get it from the History Server → app → *Download*, or from the `spark.eventLog.dir` path, e.g. `hdfs:///spark-logs/application_…`, an S3 or DBFS path, or an `eventlog_v2_*` folder for rolling logs. - **Driver or executor stderr:** from `yarn logs -applicationId <id>`, `kubectl logs`, or the Databricks/EMR log folders. - A folder that contains either kind. - `.gz` files are read directly. Files compressed with lz4, zstd or snappy need decompressing first (`zstd -d file.zstd`, `lz4 -d file.lz4`). - **Cluster shape (strongly recommended):** cores and memory per worker node, and the number of nodes. Without them the analyzer can only adjust executors relative to the current settings. - **Context:** the SLA or goal (fastest, cheapest, or just stop failing), whether the cluster is shared, and the platform (YARN, K8s, Databricks, EMR). ## 2. Run the analyzer ```bash python3 <skill-dir>/scripts/analyze_spark_log.py <logs...> \ [--node-cores 16 --node-mem-gb 64 --nodes 6] [--target-partition-mb 128] [--json] [--out report.md] ``` - `<skill-dir>` is the