palantir-core-workflow-a

Featured

Build Palantir Foundry data pipelines using Python transforms. Use when creating ETL pipelines, writing @transform decorators, or building dataset-to-dataset processing in Foundry. Trigger with phrases like "palantir pipeline", "foundry transform", "palantir ETL", "palantir data pipeline", "foundry python transform".

AI & Automation 2,926 stars 433 forks Updated yesterday MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Palantir Core Workflow A — Data Pipelines with Transforms ## Overview Build Foundry data pipelines using the `transforms-python` library. Covers the `@transform` and `@transform_df` decorators, input/output dataset wiring, incremental transforms, and `@configure` for Spark tuning. This is the primary workflow for all data processing in Foundry. ## Prerequisites - Completed `palantir-install-auth` setup - A Foundry Code Repository (Python Transforms type) - Understanding of PySpark DataFrames (Foundry runs Spark under the hood) ## Instructions ### Step 1: Project Structure ``` my-transforms-repo/ ├── src/ │ └── myproject/ │ ├── __init__.py │ ├── pipeline.py # Main transforms │ ├── utils.py # Shared logic │ └── datasets.py # Dataset path constants ├── build.gradle # Foundry build config ├── conda_recipe/meta.yaml # Dependency declarations └── settings.gradle ``` ### Step 2: Basic Transform with @transform_df ```python # src/myproject/pipeline.py from transforms.api import transform_df, Input, Output @transform_df( Output("/Company/datasets/cleaned_orders"), orders=Input("/Company/datasets/raw_orders"), ) def clean_orders(orders): """Clean raw orders: drop nulls, normalize dates, filter test data.""" from pyspark.sql import functions as F return ( orders .filter(F.col("order_id").isNotNull()) .filter(~F.col("email").like("%@test.com")) .withC...

Details

Author
jeremylongshore
Repository
jeremylongshore/claude-code-plugins-plus-skills
Created
1 years ago
Last Updated
yesterday
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category