palantir-core-workflow-a

# Palantir Core Workflow A — Data Pipelines with Transforms ## Overview Build Foundry data pipelines using the `transforms-python` library. Covers the `@transform` and `@transform_df` decorators, input/output dataset wiring, incremental transforms, and `@configure` for Spark tuning. This is the primary workflow for all data processing in Foundry. ## Prerequisites - Completed `palantir-install-auth` setup - A Foundry Code Repository (Python Transforms type) - Understanding of PySpark DataFrames (Foundry runs Spark under the hood) ## Instructions ### Step 1: Project Structure ``` my-transforms-repo/ ├── src/ │ └── myproject/ │ ├── __init__.py │ ├── pipeline.py # Main transforms │ ├── utils.py # Shared logic │ └── datasets.py # Dataset path constants ├── build.gradle # Foundry build config ├── conda_recipe/meta.yaml # Dependency declarations └── settings.gradle ``` ### Step 2: Basic Transform with @transform_df ```python # src/myproject/pipeline.py from transforms.api import transform_df, Input, Output @transform_df( Output("/Company/datasets/cleaned_orders"), orders=Input("/Company/datasets/raw_orders"), ) def clean_orders(orders): """Clean raw orders: drop nulls, normalize dates, filter test data.""" from pyspark.sql import functions as F return ( orders .filter(F.col("order_id").isNotNull()) .filter(~F.col("email").like("%@test.com")) .withC...