Methasit-Pun
UserPractical guides, prompts, and Python code for applying Anthropic's Claude Skills to data engineering and pipeline automation
Categories
Indexed Skills (23)
data-lifecycle
Umbrella skill for running a data project end-to-end through its lifecycle stages — discover sources → profile the data → architect the platform → build the medallion pipeline → refactor the code. Use this whenever the user is kicking off a new data project, asks "where do I start" or "what are the steps", or is somewhere mid-lifecycle and unsure which stage skill applies. This skill ROUTES to the stage sub-skills (data-sourcing, data-profiling, data-architecture, medallion-design, notebook-refactor) and sequences them, pulling in the right one for the user's current stage.
data-sourcing
Data collection and readiness — the step BEFORE a pipeline exists. Catalog candidate data sources for an objective, rank each by importance/impact, and record where to get it (internal system vs. open/public dataset), how to access it, its refresh cadence, licensing, and readiness blockers. Use this skill whenever a project is starting and the sources aren't decided yet, when the user asks "what data do I need for X and where do I get it", when weighing internal vs. open-source data, or when assessing whether a source is accessible/usable before committing to a build. This sits upstream of pipeline-design — it decides WHAT to ingest before deciding HOW.
data-profiling
Profile and map raw data BEFORE designing a schema. One-time exploratory analysis to learn the true shape of a dataset — row/column counts, null rates, cardinality, value distributions, ranges, data types, candidate keys, duplicates, referential relationships — then a source-to-target field mapping and an ER diagram. Use this skill whenever the user has data in hand and needs to understand it before modeling, is about to design a schema, asks "what does this data actually look like", needs to find the primary/composite key of an unfamiliar table, or must map source fields to a target model. This is exploratory and one-time; ongoing production validation is data-quality's job, and modeling patterns are schema-design's job.
medallion-design
Design a medallion (bronze/silver/gold) ETL architecture interactively, objective-first. List the available data, confirm the objective, then design GOLD first to match the objective and get the user to review it before moving down to silver, then bronze (top-down default) — or bronze-up if the user asks. Asks the user to confirm at each layer boundary rather than designing all three in one shot. Wraps the reusable utils/ library (bronze.py, silver.py, scd.py, watermark.py, metadata.py, quality.py). Use this skill whenever the user wants a bronze/silver/gold or medallion/lakehouse layout, is building a layered Delta/warehouse pipeline, or wants an objective-driven top-down layer design. For extraction/idempotency mechanics reference pipeline-design; this owns the layered design conversation.
notebook-refactor
Refactor a Jupyter/.ipynb notebook from top-to-bottom exploratory sprawl into small, named, reviewable functions with clear inputs and outputs. Extract subfunctions, separate config/params from logic, remove hidden cell-order dependencies and global-state leakage, add docstrings and light tests, and make the notebook re-runnable top-to-bottom and diff-friendly. Use this skill whenever the user has a messy notebook they want cleaned up, wants notebook code turned into functions or a module, is preparing a notebook for review/handoff/production, or asks to make an .ipynb readable or testable. This is about code STRUCTURE and reviewability; performance and correctness of data code is python-data-patterns' job.
api-design
REST API design patterns including resource naming, status codes, pagination, filtering, error responses, versioning, and rate limiting for production APIs.
backend-patterns
Backend architecture patterns, API design, database optimization, and server-side best practices for Node.js, Express, and Next.js API routes.
agent-sort
Build an evidence-backed ECC install plan for a specific repo by sorting skills, commands, rules, hooks, and extras into DAILY vs LIBRARY buckets using parallel repo-aware review passes. Use when ECC should be trimmed to what a project actually needs instead of loading the full bundle.
dbt-patterns
dbt model design, ref chains, sources, tests, macros, incremental strategies, materializations, and documentation best practices. Use this skill whenever the user is writing or reviewing dbt models, configuring dbt tests, designing model layers (staging/intermediate/marts), asking about incremental models, choosing materializations, writing macros, setting up sources.yml, or troubleshooting dbt run/test failures. Also trigger when the user mentions dbt refs, lineage graphs, model dependencies, dbt Cloud, the dbt CLI, or when they want to transform data already in the warehouse using SQL. If the project uses dbt at all, this skill should be active for any transformation questions.
python-data-patterns
Pandas, Polars, and PySpark idioms for production data engineering — chunked reads, memory-safe transforms, vectorized operations, type optimization, and performance patterns. Use this skill whenever the user is writing a Python data transformation script and running into memory issues, slow performance, or correctness bugs with large datasets. Also trigger when the user asks how to handle large CSV/Parquet files, process data in batches, use Polars instead of Pandas, optimize a PySpark job, or reduce DataFrame memory usage. If you see someone iterating row-by-row over a DataFrame, this skill should trigger immediately.
schema-design
Data modeling for analytical workloads — star schema, snowflake schema, one big table (OBT), slowly changing dimensions (SCD), normalization tradeoffs, grain definition, and surrogate key strategies. Use this skill whenever the user is designing or reviewing a data warehouse schema, planning a fact/dimension table layout, deciding how to model a business entity (customer, order, event, product), or asking how to handle historical changes to dimension attributes. Also trigger when the user asks about dbt model design, table granularity, or how to structure data for BI tools like Looker, Tableau, or Power BI. Get this right before building the pipeline — a bad schema is expensive to fix later.
sql-patterns
Best-practice SQL for analytical workloads — window functions, CTEs, query optimization, partitioning strategies, and anti-patterns to avoid. Use this skill whenever the user is writing or reviewing a SQL query that goes beyond a basic SELECT, especially on BigQuery, Snowflake, Redshift, or DuckDB. Trigger on mentions of aggregations, ranking, running totals, session analysis, lag/lead comparisons, deduplication, slowly-changing lookups, or any time the user asks "how do I write a query for X". Also trigger when a query looks slow, returns wrong results, or the user asks for a code review of existing SQL.
cloud-infra-data
AWS/GCP/Azure data infrastructure — S3/GCS/ADLS partitioning, BigQuery slot management, Redshift spectrum, Snowflake warehouses, IAM roles for data access, cost optimization, and managed service selection. Use this skill whenever the user is deploying a pipeline to cloud, choosing between managed data services, configuring storage for a data lake, setting up IAM/permissions for pipelines, asking about BigQuery pricing, Redshift vs. BigQuery vs. Snowflake, S3 bucket layout, or cloud-specific performance tuning. Also trigger when the user mentions cloud costs, slow BigQuery queries, Redshift concurrency scaling, storage formats in the cloud, or cross-account data access. If it touches cloud + data together, this skill should be active.
cost-optimization-data
Query cost analysis, partition pruning, slot reservation strategies, storage tiering, and cloud data warehouse cost reduction. Use this skill whenever the cloud data bill is unexpectedly high, a specific query is scanning too much data, the team wants to understand what's driving BigQuery/Snowflake/Redshift costs, or when choosing between on-demand vs. reserved capacity. Also trigger when the user mentions bytes scanned, slot utilization, query cost, storage costs, Redshift concurrency, Snowflake credits, or when trying to set up cost alerts and budgets. If someone says "our BigQuery bill jumped" or "this query is expensive", this skill should be active immediately.
data-migration
Moving data between systems safely — cutover planning, backfill strategies, dual-write patterns, validation, rollback procedures, and zero-downtime migration techniques. Use this skill whenever the team is migrating from one database or warehouse to another (MySQL → Snowflake, Redshift → BigQuery, on-prem → cloud), replacing a legacy pipeline, doing a major schema change on a live table, or planning a cutover that cannot have downtime. Also trigger when the user asks about dual-write, shadow reads, data validation across systems, incremental vs. full migration, or how to safely retire an old system. If the phrase "migrate", "move data", "cutover", "legacy system", or "replace the old pipeline" appears, this skill should be active.
orchestration-patterns
Airflow/Prefect/Dagster DAG design — task dependencies, retries, SLAs, backfill strategies, sensors, and failure recovery. Use this skill whenever the user is building or debugging a scheduled pipeline with multiple steps, asking how to handle task failures, setting up retries or alerts, designing a DAG structure, choosing between orchestrators, or dealing with backfill/reprocessing of historical data. Also trigger when the user mentions Airflow operators, Prefect flows, Dagster assets, task queues, or pipeline scheduling — even if they don't say "orchestration" explicitly. If a pipeline has more than two steps and needs to run on a schedule, this skill should be active.
pipeline-design
Design ETL/ELT pipelines end-to-end — source connectors, extraction strategies, transform logic, load patterns, idempotency, scheduling, and error handling. Use this skill whenever the user is starting a new ingestion job, planning how data moves from a source (REST API, database, file, webhook, message queue) into a data warehouse or data lake. Also trigger when the user asks about pipeline architecture, incremental vs. full loads, backfill strategies, CDC, retry logic, or orchestration choices (Airflow, Prefect, dbt). This skill should feel like pairing with a senior data engineer on day one of a new pipeline project.
streaming-patterns
Kafka, Flink, Kinesis, and Spark Structured Streaming design — consumer groups, partitioning, exactly-once semantics, lag monitoring, windowing, and late-arriving data. Use this skill whenever the user needs real-time or near-real-time data processing, is redesigning a batch pipeline into streaming, asks about event-driven architectures, or mentions Kafka topics, consumer lag, checkpointing, watermarks, or stream-table joins. Also trigger when the user says batch is "too slow", stakeholders want "live" dashboards, or the pipeline needs to react to events as they happen rather than on a schedule. If latency requirements are under a few minutes, this skill should be active.
data-contracts
Define and enforce schema contracts between producer and consumer teams — field types, nullability, allowed values, versioning, breaking vs. non-breaking changes, and change detection patterns. Use this skill whenever two teams or services share a dataset and upstream changes keep breaking the downstream silently, when a team wants to formalize what a dataset "promises" to its consumers, or when setting up schema validation at pipeline boundaries. Also trigger when the user asks about schema evolution, backward/forward compatibility, schema registries, Great Expectations for inter-team contracts, or when a producer is about to make a "small" schema change and you want to assess its downstream impact. If upstream and downstream are owned by different people, this skill should be active.
data-governance
Data lineage tracking, PII tagging, access control policies, data catalog metadata standards, retention policies, and audit logging for regulatory compliance. Use this skill whenever the company is subject to PDPA, GDPR, HIPAA, or any data privacy regulation, when an audit requires proof of who accesses what data, when PII fields need to be identified and classified in a dataset, when setting up column-level access control, or when building a data catalog. Also trigger when someone asks about data masking, anonymization, right-to-erasure workflows, role-based data access, or data lineage from source to BI tool. If the word "compliance", "audit", "PII", "sensitive data", or "regulation" appears, this skill should be active.
data-quality
Write systematic data quality checks — validation rules, Great Expectations suites, dbt tests, anomaly detection, null/type/range/referential integrity assertions, and monitoring patterns for production pipelines. Use this skill whenever the user is dealing with bad data in a pipeline, setting up validation before or after a load step, adding tests to dbt models, writing Great Expectations expectations, or trying to detect when upstream data has changed shape. Also trigger when stakeholders keep finding incorrect numbers, when a pipeline silently loads garbage, or when the user asks "how do I make sure my data is correct". Prevention is cheaper than debugging.
ml-feature-engineering
Feature store patterns, training/serving skew prevention, feature pipelines for ML teams, point-in-time correct joins, and bridging data engineering with MLOps conventions. Use this skill whenever an ML team needs feature pipelines, when building a feature store or deciding whether to use one, when there's a training/serving skew problem (model performance in production differs from validation), when features need to be shared across multiple models, or when designing point-in-time correct feature computation. Also trigger when the user mentions feature stores (Feast, Tecton, Hopsworks), label leakage, backfilling features, offline/online store separation, or when data engineering work feeds directly into model training. If the words "features", "training set", "model pipeline", or "MLOps" appear alongside data engineering questions, this skill should be active.
stakeholder-reporting
Translate pipeline metrics, SLA breaches, data quality failures, and incidents into clear non-technical summaries for business stakeholders. Use this skill whenever data was late, wrong, or missing and someone needs to communicate what happened to a manager, director, or business team. Also trigger when writing incident reports, SLA breach notifications, data quality summaries, pipeline health updates, or any communication where the audience doesn't know what Airflow or BigQuery is. If the user needs to explain a technical data failure to a non-technical person, this skill should be active.
Bio shown is the top-scored skill's repo description as a fallback — real GitHub bios land in a future update.