market-data-engineeringlisted

# Market data engineering The layer under every backtest, and the one where corruption is silent. Most of what follows was **executed**, not read from docs. ## 1. The findings that will change your code | # | Finding | |---|---| | 1 | 🚨 **`polars` is now an empty shim** — the wheel is `py3-none-any`, ~865 KB, **no compiled code**. It hard-depends on `polars-runtime-32==<same version>` (~50 MB, `cp310-abi3`). Split landed at 1.34.0b2 (2025-09-26). **A lockfile listing only `polars` does not pin the engine.** `polars-lts-cpu` froze at 1.33.1, the release before. **2.0.0rc1 shipped 2026-09-02.** | | 2 | 🚨 **`polars.join_asof` does NOT check sortedness when you pass `by=`** — its own docstring: the in-memory engine *"cannot check the sortedness if 'by' groups are provided"*. Unsorted input → **silently wrong rows, no error**. That is the exact per-symbol quote join everyone writes | | 3 | ✅ **`pandas.merge_asof` DOES raise** `ValueError: left keys must be sorted`. **The "pandas silently gives garbage" folklore is out of date** — polars is now the unsafe one | | 4 | 🚨 **pandas and polars disagree on what "sorted" means.** pandas wants a **global** sort by `on`; polars wants sorted **within each `by` group**. Porting code between them silently changes results | | 5 | 🚨 **pandas 3.0 (2026-01-21) defaults to `datetime64[us]`, not `[ns]`.** Any `.astype("int64")` on timestamps now yields integers **1000× smaller** | | 6 | 🚨 **float64 cannot hold epoch nanoseconds.** At 2024 ti