databricks-data-handling

# Databricks Data Handling ## Overview Implement GDPR compliance, PII masking, data retention, and row-level security in Delta Lake with Unity Catalog. Covers data classification tagging, right-to-deletion workflows, automated retention enforcement, column-level masking functions, and subject access request (SAR) reporting. ## Prerequisites - Unity Catalog enabled - Understanding of data classification requirements (GDPR, CCPA, HIPAA) - Admin access for tags and masking functions ## Instructions ### Step 1: Classify and Tag Data Use Unity Catalog tags to classify tables and columns for automated compliance enforcement. ```sql -- Tag tables with classification and retention ALTER TABLE prod_catalog.silver.customers SET TAGS ('data_classification' = 'PII', 'retention_days' = '730'); ALTER TABLE prod_catalog.silver.orders SET TAGS ('data_classification' = 'CONFIDENTIAL', 'retention_days' = '365'); ALTER TABLE prod_catalog.gold.metrics SET TAGS ('data_classification' = 'INTERNAL', 'retention_days' = '1825'); -- Tag PII columns ALTER TABLE prod_catalog.silver.customers ALTER COLUMN email SET TAGS ('pii_type' = 'email'); ALTER TABLE prod_catalog.silver.customers ALTER COLUMN phone SET TAGS ('pii_type' = 'phone'); ALTER TABLE prod_catalog.silver.customers ALTER COLUMN full_name SET TAGS ('pii_type' = 'name'); ``` ### Step 2: GDPR Right-to-Deletion Delete all user data across PII-tagged tables with audit logging. ```python from pyspark.sql import SparkSession from datetim...