handling-schema-evolutionlisted

# Handling Schema Evolution ## When to use - Changing a table, file, or event schema that others read. - Adding, renaming, dropping, or retyping columns. - Choosing a compatibility mode for Avro/Iceberg/Delta/Parquet. - Do NOT use for cross-team producer interfaces (use `designing-data-contracts`). ## Compatibility model - **Backward compatible** — new readers can read old data (safe to add optional fields with defaults). Most warehouse evolution targets this. - **Forward compatible** — old readers can read new data (they ignore new fields). - **Full** — both. Aim for backward-compatible-by-default. ## Workflow ``` - [ ] Classify the change: additive (safe) or breaking - [ ] Prefer additive: add nullable/defaulted columns - [ ] For renames/type changes, add-new + backfill + dual-write, then deprecate - [ ] Enable the format's schema evolution settings deliberately - [ ] Communicate + version breaking changes ``` 1. **Classify.** Additive (new optional column) is safe. Rename, drop, type narrowing, or nullability tightening are breaking. 2. **Prefer additive.** Add a nullable/defaulted column instead of mutating an existing one. 3. **Migrate breaking changes in steps**: add the new column, backfill it, dual-write old+new, switch readers, then drop the old column later. 4. **Configure the format** — evolution is opt-in and format-specific (below). 5. **Version + announce** anything breaking. ## Patterns **Additive, backward-compatible column:** ```sql ALTER