csv-cleanerlisted

# CSV Cleaner Two-phase workflow: diagnose first, clean second. Never start fixing before knowing what is broken. ## Phase 1: Diagnose Run `scripts/diagnose.py` on the input file: ```bash python scripts/diagnose.py <path-to-file> ``` The script reports: - Encoding (utf-8, latin-1, utf-16, etc.) - Delimiter (comma, tab, semicolon) - Quote character and escape behavior - Header row detection (present, absent, or duplicate) - Row count - Column count consistency across rows - Type inference per column (integer, float, date, string, mixed) - Null distribution per column - Duplicate row count Read the output carefully before doing anything else. The diagnosis tells you which fixes apply. ## Phase 2: Clean Apply fixes in this order: 1. Fix encoding: re-read with detected encoding, write as utf-8. 2. Normalize line endings to `\n`. 3. Strip BOM and zero-width characters. 4. Fix headers: lowercase, snake_case, replace spaces with underscores, ensure uniqueness. 5. Trim whitespace from every cell. 6. Coerce types where the column is type-consistent. 7. Standardize null representations: empty string, "NA", "null", "n/a" all become empty. 8. Deduplicate exact rows. 9. Validate column count consistency. Flag rows that do not match the header. Do not silently change data. If a fix alters values, report the count. ## Output Save the cleaned file as `<original-name>-cleaned.csv` in the same directory. Always utf-8, always comma-delimited, always with a header row. Then emit a