model-validation
FeaturedDesign or audit the clinical-validation study for an engineer-built medical-imaging model (segmentation, classification, or detection) before the validation report or manuscript is written. Covers patient-level split disjointness and the data-leakage taxonomy, tuning-on-test, internal versus genuine external validation, comparator design, single-run versus multi-seed variance, task-correct metric selection, test-set sizing, and CLAIM 2024 / TRIPOD+AI / STARD-AI reporting fit. Ships a deterministic split-leakage gate that proves patient disjointness by set arithmetic on the emitted split-assignment table. Does not build or train models — it integrates with MONAI / nnU-Net, it does not replace them.
Install
Quality Score: 95/100
Skill Content
Details
- Author
- Aperivue
- Repository
- Aperivue/medsci-skills
- Created
- 3 months ago
- Last Updated
- today
- Language
- Python
- License
- MIT
Bundled in these plugins
Similar Skills
Semantically similar based on skill content — not just same category
model-evaluation
Compute and report task-correct held-out metrics for a trained medical-imaging model — segmentation (Dice plus a boundary metric such as HD95 or NSD, per structure), classification (AUROC plus AUPRC and sensitivity/specificity with bootstrap CIs at the deployment prevalence), detection (FROC or mAP with a stated IoU criterion), interactive/promptable segmentation (the interaction-count, convergence, and per-case-time axes a static Dice omits), or generative/synthesis image evaluation (similarity plus the downstream-task efficacy similarity alone cannot establish) — plus calibration and subgroup slices. Emits a per-case results table that analyze-stats turns into publication tables, and gates the metric choice against Metrics Reloaded, CLAIM 2024, and Park et al. 2024 (no pixel accuracy for segmentation, no bare accuracy under imbalance, no static Dice for an interactive method, no similarity-only claim for a generative model). Numbers come only from executed code, never hand-typed.
model-scaffold
Generate a reproducible, runnable PyTorch training repo for a medical-imaging task — segmentation, classification, detection, image-to-image synthesis, self-supervised pretraining, or fine-tuning a pretrained backbone (transfer learning) — the missing middle link between choosing an architecture and validating a trained model. Emits a patient-level seed-locked split as an auditable artifact, a task-appropriate model, train and evaluate scripts that seed every RNG and infer under eval mode, a config, requirements, a reproducibility record, and a Methods stub with VERIFY placeholders (no fabricated numbers). Fine-tuning mode adds a frozen-then-unfrozen schedule, discriminative learning rates, and a pretrained-weight provenance record. The reproducibility guarantees hold by construction, so the build is leakage-safe before any training runs. Integrates with MONAI, nnU-Net, TorchIO, timm, and torchvision — it does not reimplement them.
mllm-eval
Design or audit a model-agnostic evaluation harness for an LLM or multimodal LLM on a clinical task (radiology report generation, visual question answering, clinical text extraction/classification) — the adjudicated reference standard, clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 beyond BLEU/ROUGE), faithfulness and hallucination, pretraining-contamination of public benchmarks, prompt-sensitivity and determinism, answer-matching, and a reader study — and gate the plan for those axes. Works on a closed API or open weights. Never fabricates outputs or scores, and never reports n-gram overlap as clinical correctness.