covariance-and-risk-modelslisted

Estimate a covariance matrix an optimizer can actually invert, and report how much variance it hides. TRIGGER - covariance matrix estimation, sample covariance singular, "matrix is not positive definite", np.cov more assets than observations, N > T, condition number, Ledoit-Wolf shrinkage, sklearn LedoitWolf, CovarianceShrinkage, shrinkage intensity or delta, RiskMetrics EWMA, lambda 0.94 or 0.97, exponentially weighted covariance, exp_cov span, PCA or statistical factor risk model, Marchenko-Pastur, Barra fundamental factor model, specific risk, predicted vs realized volatility, risk model bias test; "my minimum-variance portfolio has 90x leverage", "the optimizer says 0% risk". SKIP for turning a covariance into weights and the optimizers themselves (portfolio-optimizers), for VaR, Expected Shortfall and their backtests (risk-measures-var-cvar), for GARCH and univariate volatility forecasting (volatility-models), and for Sharpe and drawdown conventions (portfolio-and-risk).
howard-lynn-ye/fin-skills · ★ 1 · AI & Automation · score 77

Install: claude install-skill howard-lynn-ye/fin-skills

# Covariance and risk models **`np.cov(returns.T)` is an unbiased estimate of every entry and a bad estimate of the matrix.** The optimizer does not consume entries; it consumes the inverse, and the inverse is where the estimation error concentrates. Two failures follow, and both look like success in-sample: with N > T the matrix is singular and the optimizer finds its null space, and even with N < T the variance it predicts for its own portfolio is biased low. Every number below is printed by `scripts/risk_model.py` (numpy / pandas, fixed seeds, about 10 s). The data is a k-factor DGP whose true covariance is known, so "predicted", "true" and "out-of-sample realized" are three separate columns rather than two. ## 1. 🚨 N > T: the optimizer reports 0.00 % risk and delivers 243 % ✅ Measured: 100 assets from a 3-factor DGP, minimum-variance weights `Sigma^-1 1 / 1'Sigma^-1 1`, volatilities annualized at sqrt(252), `true` from the DGP covariance: | T | estimator | rank | cond | gross lev | max abs w | pred vol | true vol | true/pred var | |---|---|---|---|---|---|---|---|---| | **60** | 🚨 sample | **59** | 2.25e+20 | **92.5** | 2.92 | **0.00 %** | **243.44 %** | **-9.0e+15** | | 60 | Ledoit-Wolf identity | 100 | 1.82e+02 | 3.2 | 0.09 | 4.92 % | 10.67 % | 4.69 | | 250 | sample | 100 | 5.39e+02 | 3.5 | 0.16 | 5.13 % | 9.05 % | 3.11 | | 250 | Ledoit-Wolf identity | 100 | 3.30e+02 | 3.2 | 0.15 | 5.59 % | 8.61 % | 2.38 | | 1000 | sample | 100 | 2.19e+02 | 2.6 | 0.20 | 6.52 % |