pandas

# Pandas ## Purpose Transform and analyze tabular data correctly and at speed. Pandas makes it easy to write code that is slow, and easier still to write code that is silently wrong. ## When to Use - Cleaning, transforming, or analyzing tabular data in Python. - A pandas operation that is slow or exhausting memory. - Reviewing analysis code for correctness. - Deciding whether the dataset has outgrown pandas. ## Capabilities - Vectorized operations and eliminating row-wise loops. - Memory reduction through dtype selection. - Merge and join semantics, including the ones that silently duplicate rows. - Groupby, aggregation, and window functions. - Chunked processing and the migration path to Polars or DuckDB. ## Inputs - The data source, its size, and its schema. - The transformation or analysis required. - The memory available. ## Outputs - Vectorized transformations with no `iterrows`. - Explicit dtypes, including categoricals for low-cardinality strings. - Joins with verified cardinality. ## Workflow 1. **Set dtypes at read time** — Reading a CSV without `dtype` gives you `object` columns and `float64` for everything numeric. This is usually a 5-10x memory difference. 2. **Vectorize** — Any `for` loop or `iterrows` over a DataFrame should be a vectorized expression, a `groupby`, or a `merge`. `apply` is a loop with better syntax. 3. **Verify every join** — `merge(..., validate="one_to_many")`. An unvalidated join that is secretly many-to-many silently multiplies y...