large-file-data-analysislisted
Install: claude install-skill shali10/awesome-agent-skills
# Large File & Spreadsheet Data Analysis Engine
## Overview
High-performance data analysis skill tailored for AI Agents. It provides streaming ingestion pipelines via `openpyxl(read_only=True)` and automatic Parquet conversion, enabling memory-safe querying of 100,000+ row datasets in constant <50MB RAM.
## When to Use & When NOT to Use
### When to Use
- Excel spreadsheets (`.xlsx`/`.xls`) with ≥10,000 rows or multiple dense sheets.
- Multi-megabyte PDF, Word, or CSV datasets requiring statistical aggregation.
- Generating schema summaries and targeted slices without loading raw data into LLM context.
### When NOT to Use
- Small tables (<100 rows) where standard markdown tables or direct pandas reads are fine.
- Visual chart rendering without data extraction (use image/visualization tools).
## Streaming Extraction Pattern
```python
import openpyxl
def stream_excel_summary(file_path: str, sample_rows: int = 5):
wb = openpyxl.load_workbook(filename=file_path, read_only=True, data_only=True)
summary = {}
for sheet in wb.sheetnames:
ws = wb[sheet]
rows = ws.iter_rows(values_only=True)
try:
header = next(rows)
except StopIteration:
continue
row_count = 0
samples = []
for row in rows:
row_count += 1
if len(samples) < sample_rows:
samples.append(row)
summary[sheet] = {
"columns": header,