apache-spark-optimizer
SolidAnalyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization
Install
Quality Score: 91/100
Skill Content
Details
- Author
- a5c-ai
- Repository
- a5c-ai/babysitter
- Created
- 10 months ago
- Last Updated
- 2 weeks ago
- Language
- JavaScript
- License
- MIT
Similar Skills
Semantically similar based on skill content — not just same category
spark-optimizer
Analyse Apache Spark logs and recommend the best Spark configuration. Use when the user shares or points to a Spark event log, History Server download, eventlog_v2 folder, or driver/executor stderr (YARN, Kubernetes, EMR, Databricks, Dataproc). Also use when they ask to tune or speed up a Spark job, or to fix OOM, "Container killed by YARN", spill, skew, slow stages, GC or shuffle fetch failures, or to size executors, memory, cores and spark.sql.shuffle.partitions.
spark-sql-optimizer
Optimize spark sql optimizer operations. Auto-activating skill for Data Pipelines. Triggers on: spark sql optimizer, spark sql optimizer Part of the Data Pipelines skill category. Use when working with spark sql optimizer functionality. Trigger with phrases like "spark sql optimizer", "spark optimizer", "spark".
optimizing-pyspark-jobs
Optimize slow or failing PySpark and Spark SQL jobs — partitioning and repartitioning, data skew, shuffles, broadcast joins, caching, Adaptive Query Execution, and avoiding driver collects and Python UDFs. Use when a Spark job is slow, spills, OOMs, has skewed tasks, runs a huge shuffle, or a stage hangs on a few straggler tasks.