data-quality

# Data Quality ## Purpose Catch bad data before it reaches a dashboard, a model, or a customer. A pipeline that silently propagates corrupt data is worse than one that fails, because the failure is discovered downstream, later, by someone who trusts the number. ## When to Use - Ingesting data from a source you do not control. - Building quality gates into a pipeline. - Investigating a metric that looks wrong. - Auditing a dataset before it is used for analysis or training. ## Capabilities - Profiling: distributions, cardinality, null rates, outliers. - Schema validation and type enforcement. - Constraint checks: uniqueness, referential integrity, ranges, formats. - Freshness, completeness, and volume anomaly detection. - Quarantine and alerting patterns. ## Inputs - The dataset and its expected schema. - The business rules the data must satisfy. - Historical volume and distribution, for anomaly baselines. ## Outputs - A profile of the data as it actually is, not as documented. - Validation checks that run on every load. - A quarantine path for rows that fail, and an alert when they do. ## Workflow 1. **Profile before you trust** — Row count, null rate, cardinality, min/max, and the distribution of every column. The documented schema and the actual data disagree more often than not. 2. **Validate the schema at the boundary** — Column presence, types, and nullability, checked on ingest. A silently added column or a type change upstream is the most common pipeline br...