exploring-data

# Exploring Data ## 0. Route by size FIRST ```bash ls -la <filepath> # or: wc -l for row estimate ``` - **< 200MB and < ~5M rows** → ydata-profiling path (section A). Exact stats, interactive HTML. - **Larger** → large-file path (section B). ydata-profiling loads everything into pandas and will crawl or OOM; the DuckDB/sketch path runs in fixed memory at any size. - **Task-specific ops** (any size): duplicates, join feasibility, drift → section C. ## A. Standard path (ydata-profiling) ### 1. Check if installed (instant) ```bash bash /mnt/skills/user/exploring-data/scripts/check_install.sh ``` Returns: `installed` or `not_installed` ### 2. Install if needed (one-time, ~19s) ```bash if [ "$(bash /mnt/skills/user/exploring-data/scripts/check_install.sh)" = "not_installed" ]; then bash /mnt/skills/user/exploring-data/scripts/install_ydata.sh fi ``` ### 3. Run analysis (always generates JSON + HTML by default) ```bash bash /mnt/skills/user/exploring-data/scripts/analyze.sh <filepath> [minimal|full] [html|json] ``` **Defaults:** minimal + html (also generates JSON) **Output:** - `eda_report.html` - Interactive report for user - `eda_report.json` - Machine-readable for Claude analysis ### 4. If Claude needs to analyze (user asks "what do you think?" etc.) ```bash python /mnt/skills/user/exploring-data/scripts/summarize_insights.py /mnt/user-data/outputs/eda_report.json ``` Claude should read the stdout markdown summary, NOT the full JSON report. ### 5. Present find...