evidently-monitoringlisted
Install: claude install-skill testland/qa-skills
# evidently-monitoring
Evidently is *"an open-source Python library with over 40+ million
downloads. It provides 100+ evaluation metrics, a declarative testing
API, and a lightweight visual interface"* per [Evidently docs].
## When to use
- Pre-deployment gate: assert no data/target drift between
candidate-model evaluation set and the production reference.
- Production monitoring: scheduled job comparing yesterday's traffic
vs the reference window.
- Triage tool: when a model misbehaves in prod, run an Evidently
Report comparing the bad period to a known-good window.
## How to use
1. `pip install evidently` into the CI or monitoring environment (Step 1).
2. Load a pinned reference dataset and the current dataset, and wrap both
in a `Dataset` with the same `DataDefinition` (Step 2).
3. Run `Report([DataDriftPreset()])` with
`run(reference_data=, current_data=)` and `save_html` for review (Step 3).
4. For a CI gate, add `include_tests=True`, read `.dict()["tests"]`, and
`raise SystemExit` on any `FAIL`/`ERROR` (Step 4).
5. For performance, map the task in the data definition (`BinaryClassification`
with `target`, `prediction_probas` and `pos_label`) and set the business
`probas_threshold` before trusting precision or recall (Step 5).
6. Wrap the gated Report in a daily scheduled job comparing yesterday's
traffic to the pinned reference, notifying on-call on failure (Step 6).
7. Tune the per-column drift method (`psi`, etc.) and severity tiers so