palantir-cost-tuning

# Palantir Cost Tuning ## Overview Optimize Foundry compute and API costs through incremental transforms, right-sized Spark profiles, efficient pagination, and usage monitoring. ## Prerequisites - Active Foundry enrollment with build history - Access to Foundry resource usage metrics - Understanding of transform build patterns ## Instructions ### Step 1: Cost Drivers in Foundry | Cost Category | Driver | Optimization | |---------------|--------|-------------| | Compute | Full rebuilds of large transforms | Use `@incremental()` | | Compute | Oversized Spark profiles | Right-size `@configure` profiles | | Storage | Redundant dataset snapshots | Configure retention policies | | API | High-frequency polling | Use webhooks instead | | API | Small page sizes | Use max page_size (500) | ### Step 2: Convert Full Rebuilds to Incremental ```python from transforms.api import transform_df, Input, Output, incremental # BEFORE: Full rebuild every run (expensive for large datasets) @transform_df(Output("/out"), data=Input("/in")) def expensive(data): return data.filter(data.status == "active") # AFTER: Only processes new/changed rows @incremental() @transform_df(Output("/out"), data=Input("/in")) def cheap(data): return data.filter(data.status == "active") ``` ### Step 3: Right-Size Spark Profiles ```python from transforms.api import configure # DON'T: Default profile for everything # DO: Match profile to actual data size # Small data (< 1GB) — use lightweight transforms (...