← ClaudeAtlas

streaming-into-data-lakelisted

Stream rows continuously into Apache Iceberg tables on S3 Tables (or standard Iceberg on a general purpose bucket) using Amazon Data Firehose with IcebergDestinationConfiguration. Covers the Firehose stream, the optional record-shaping Lambda (ProcessingConfiguration), the Lake Formation grants required for Firehose to write through Glue, and the three-phase CloudFormation deploy that gates the stream on grants existing. Triggers on: stream into data lake, Firehose to S3 Tables, Firehose to Iceberg, real-time ingestion to Iceberg, Firehose IcebergDestinationConfiguration, stream Kinesis to Iceberg, near real-time CDC to data lake, continuous append to Iceberg. Do NOT use for: batch file imports (use ingesting-into-data-lake), creating the destination Iceberg table itself (use creating-data-lake-table), the one-time Lake Formation onboarding (use setting-up-lake-formation), finding tables by fuzzy name (use finding-data-lake-assets), running queries (use querying-data-lake), Glue connections (Firehose does not
jaingxyz/aws-data-skills · ★ 0 · DevOps & Infrastructure · score 75
Install: claude install-skill jaingxyz/aws-data-skills
<!-- A submission-flavored copy of this skill (with the extra deployment metadata the AWS MCP server registry requires) lives in a separate fork. The version here is the public learning copy. --> # Stream into Data Lake (Firehose to Iceberg) Continuously land rows into an Iceberg table on S3 Tables or a general purpose bucket using Amazon Data Firehose. This skill assumes the destination Iceberg table already exists. Creating the table is owned by `creating-data-lake-table`. The first-time Lake Formation onboarding is owned by `setting-up-lake-formation`. ## Why this skill exists Wiring Firehose to Iceberg through Lake Formation hits four failure modes that no AWS doc surfaces in one place: (1) Firehose silently routes records to the error bucket when the Lambda transformer's output column names do not exactly match the Iceberg schema; (2) Firehose drops Iceberg writes with `AccessDenied` until the Firehose role holds `DESCRIBE` and `SELECT` on the catalog and database plus `SELECT, INSERT, DELETE` on the table; (3) defining the Firehose stream and the Lake Formation grants in one CloudFormation template creates a race because CloudFormation evaluates IAM at stack-create time, before the grants apply; (4) re-deploying the stack with Firehose conditionals enabled but other condition flags omitted resets the unspecified flags to their template default of `false`, silently disabling resources that were already live. This skill encodes the working sequence and the precise er