information-fill-the-whole-results-grid

Solid

Use at study design and again at writing when the source reports a grid — variants crossed with backbones, datasets or metrics — and you are about to fill part of it. Covers reproducing the whole grid at reduced N where you must, and why a labelled reduced-N cell beats an empty one.

AI & Automation 804 stars 25 forks Updated today NOASSERTION

Install

View on GitHub

Quality Score: 82/100

Stars 20%
97
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Reproduce the whole (variant x backbone x dataset x metric) grid, reduced-N where you must The contribution of an AI/ML systems paper is a grid: method variants x backbones or base models x datasets x metric families, plus one ablation per named component and the qualitative demonstrations. Completeness of the grid is audited; nothing pays for extra depth in one cell. At design time, write the grid out as a table of empty cells and schedule the cheapest run that fills each one. Widening is additive, not a substitution: the item the task actually ships keeps its own named subsection with its own values, in the source's units, even when the full grid gives a tighter interval. A run that priced the two scopes honestly, chose the fifteen-paper corpus, and left the one shipped paper as an appendix row scored 5/15/5 where an agent that simply printed the shipped paper's own result scored 32/25/45. See `the-supplied-item-is-the-graded-unit`. Treat a single supplied example file as a smoke-test fixture for the *grid*, not as the evaluation set and not as something the report may drop: obtain the released implementation, the pretrained weights, the full benchmark suite and the baseline systems from the public release. When a cell cannot be run at full scale, run it at reduced N or one seed and label it as such. A crude arm counts; a Limitations sentence declaring the arm out of scope reads as the experiment never having been attempted. Ablate each named component one at a time a...

Details

Author
tangxiangru
Repository
tangxiangru/AutoR
Created
5 months ago
Last Updated
today
Language
Python
License
NOASSERTION

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

earth-the-technique-grid-needs-values-in-its-cells

Use at study design when the figure list is chosen, at implementation before any pipeline code is written, and again at analysis, when the supplied archive is stratified by measurement technique or instrument and you are about to describe that stratification. Covers reading the per-technique columns the archive already ships, drawing the technique grid with estimates in it rather than file counts, and why a no-peek rule about the reference product must not reach your figures.

804 Updated today
tangxiangru
AI & Automation Solid

chemistry-reproduce-the-scoring-path-before-you-replace-it

Use at implementation, experimentation and analysis when you are reproducing a published benchmark number and the source's scoring path is one you can read — which rows are scored, in what order, how many the loader drops, which epoch is reported, how tasks are pooled, over how many seeds. Covers implementing that path exactly before improving it, the one-row-per-step ladder from the published rule down to your own honest estimate, and why one un-replicated step makes the reproduction gap you report uninterpretable.

804 Updated today
tangxiangru
AI & Automation Solid

life-full-study-skeleton-including-the-wet-lab-half

Use at study design and again when laying out the results section, to check every slot of a life-science study is filled. Covers the skeleton a paper of this kind carries, and what to put in the slots this run cannot compute rather than leaving them out.

804 Updated today
tangxiangru