model-evaluation
FeaturedCompute and report task-correct held-out metrics for a trained medical-imaging model — segmentation (Dice plus a boundary metric such as HD95 or NSD, per structure), classification (AUROC plus AUPRC and sensitivity/specificity with bootstrap CIs at the deployment prevalence), detection (FROC or mAP with a stated IoU criterion), interactive/promptable segmentation (the interaction-count, convergence, and per-case-time axes a static Dice omits), or generative/synthesis image evaluation (similarity plus the downstream-task efficacy similarity alone cannot establish) — plus calibration and subgroup slices. Emits a per-case results table that analyze-stats turns into publication tables, and gates the metric choice against Metrics Reloaded, CLAIM 2024, and Park et al. 2024 (no pixel accuracy for segmentation, no bare accuracy under imbalance, no static Dice for an interactive method, no similarity-only claim for a generative model). Numbers come only from executed code, never hand-typed.
Install
Quality Score: 95/100
Skill Content
Details
- Author
- Aperivue
- Repository
- Aperivue/medsci-skills
- Created
- 3 months ago
- Last Updated
- today
- Language
- Python
- License
- MIT
Bundled in these plugins
Similar Skills
Semantically similar based on skill content — not just same category
model-validation
Design or audit the clinical-validation study for an engineer-built medical-imaging model (segmentation, classification, or detection) before the validation report or manuscript is written. Covers patient-level split disjointness and the data-leakage taxonomy, tuning-on-test, internal versus genuine external validation, comparator design, single-run versus multi-seed variance, task-correct metric selection, test-set sizing, and CLAIM 2024 / TRIPOD+AI / STARD-AI reporting fit. Ships a deterministic split-leakage gate that proves patient disjointness by set arithmetic on the emitted split-assignment table. Does not build or train models — it integrates with MONAI / nnU-Net, it does not replace them.
mllm-eval
Design or audit a model-agnostic evaluation harness for an LLM or multimodal LLM on a clinical task (radiology report generation, visual question answering, clinical text extraction/classification) — the adjudicated reference standard, clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 beyond BLEU/ROUGE), faithfulness and hallucination, pretraining-contamination of public benchmarks, prompt-sensitivity and determinism, answer-matching, and a reader study — and gate the plan for those axes. Works on a closed API or open weights. Never fabricates outputs or scores, and never reports n-gram overlap as clinical correctness.
explainability
Produce or audit the interpretability/explainability analysis of a medical-imaging model — Grad-CAM / Grad-CAM++ / attention-rollout / saliency / integrated-gradients — so it clears the rigor bar a reviewer expects: mandatory Adebayo sanity checks (model- and data-randomisation), a quantitative localisation metric against ground truth (IoU / pointing game / Dice) instead of eyeballed examples, a cohort-level result rather than cherry-picked cases, and attribution framing rather than "proof the model is correct". Emits an explainability-report manifest and a deterministic rigor gate. Integrates captum / pytorch-grad-cam; it does not reimplement them, and never runs a model on real patient data.