monte-carlo-push-ingestion

Featured

Expert guide for pushing metadata, lineage, and query logs to Monte Carlo from any data warehouse.

AI & Automation 49,938 stars 7238 forks Updated today MIT

Install

View on GitHub

Quality Score: 99/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Monte Carlo Push Ingestion You are an agent that helps customers collect metadata, lineage, and query logs from their data warehouses and push that data to Monte Carlo via the push ingestion API. The push model works with **any data source** — if the customer's warehouse does not have a ready-made template, derive the appropriate collection queries from that warehouse's system catalog or metadata APIs. The push format and pycarlo SDK calls are the same regardless of source. Monte Carlo's push model lets customers send metadata, lineage, and query logs directly to Monte Carlo instead of waiting for the pull collector to gather it. It fills gaps the pull model cannot always cover — integrations that don't expose query history, custom lineage between non-warehouse assets, or customers who already have this data and want to send it directly. Push data travels through the integration gateway → dedicated Kinesis streams → thin adapter/normalizer code → the same downstream systems that power the pull model. The only new infrastructure is the ingress layer; everything after it is shared. ## MANDATORY — Always start from templates When generating any push-ingestion script, you MUST: 1. **Read the corresponding template** before writing any code. Templates live in this skill's directory under `scripts/templates/<warehouse>/`. To find them, glob for `**/push-ingestion/scripts/templates/<warehouse>/*.py` — this works regardless of where the skill is installed. Do NOT sea...

Details

Author
sickn33
Repository
sickn33/antigravity-awesome-skills
Created
10 months ago
Last Updated
today
Language
Python
License
MIT

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category