← ClaudeAtlas

model-performance-regression-gatelisted

Use when a retrained ML model is a candidate for promotion and CI must block it if it got worse. Compares held-out accuracy, F1, AUC or RMSE with the production model within a tolerance, and checks key segments with Deepchecks so a global gain cannot hide a slice regression.
testland/qa-skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill testland/qa-skills
# model-performance-regression-gate A CI gate that blocks model promotion when a retrained model regresses on held-out metrics vs the current production model beyond a configured tolerance. Covers global metrics (accuracy, F1, AUC, RMSE) and per-segment checks so a model that improves in aggregate but regresses on a key slice is still blocked. Differentiation from neighbors: - `deepchecks-tests` runs the full model-evaluation suite but does not compare a candidate against a production model or enforce per-metric tolerances as a promotion gate. - `evidently-monitoring` detects data and target drift in production traffic; it does not perform a pairwise metric comparison between two model versions at deploy time. - This skill is scoped to the promotion-gating decision: candidate vs. production, per metric, per segment, with configurable tolerances. ## When to use Invoke in a CI/CD pipeline step immediately after retraining, before the model artifact is registered or deployed. The step receives the held-out test set, the candidate model, and the production model (loaded from the registry). It exits non-zero when any metric degresses beyond tolerance. ## Step 1 - Install dependencies ```bash pip install deepchecks scikit-learn joblib ``` Deepchecks is the primary framework for segment-level checks (per [Deepchecks model evaluation docs]). scikit-learn supplies the scalar metric functions (per [scikit-learn model evaluation docs]). ## Step 2 - Load models and held