copy-the-graders-limits-not-just-its-logic

Solid

Use at study design, implementation and experimentation when the task's score is produced by *executing* what you submit - running generated programs against hidden test cases, simulating, decoding, solving, rendering - and where you are building a local copy of the scorer to choose among candidates or to measure a method before committing to it. Covers reading the resource limits out of the shipped scorer's source rather than out of the prose that summarises it, giving every limit a named constant with its source line beside it, why a replica looser than the grader is far worse than one that is stricter, the failure taxonomy your local report must be able to express (a bucket that is empty in every arm is the tell), and the item-by-item calibration that catches a replica whose average already agrees.

AI & Automation 805 stars 25 forks Updated 2 weeks ago NOASSERTION

Install

View on GitHub

Quality Score: 82/100

Stars 20%
97
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# When the score comes from a run, the limits are part of the metric Most graded tasks compute a number from your file. Some *execute* it: they run your program, your simulation, your solver, your decode, under a harness with a clock and a memory cap, and turn what happens into the score. On those tasks the metric is not a formula - it is a runtime environment, and its resource limits are as much a part of the definition as the comparison at the end. A local replica of that harness is the right thing to build; you cannot choose among candidates without one. But a replica inherits the logic easily and the limits almost never, because the logic is what the code obviously does and the limits are constants sitting three call frames away. ## Enumerate the limits from the source, then check the prose against it Briefs of this kind state the limits twice: a prose section - "Resource Limits", "Execution Environment", "Evaluation protocol" - and the scorer's own source, often pasted into the same document or sitting in the task directory. The prose is a summary and it is incomplete, in the same way and for the same reason that a stated submission shape is (`the-row-count-comes-from-the-split-not-the-brief`). What is specific here is the **direction** of the error. Every limit in the source that the prose does not mention is a place where you will invent one instead, and an invented budget is always the loose one: nobody guesses a cap stricter than the cap they were told about. So...

Details

Author
tangxiangru
Repository
tangxiangru/AutoR
Created
6 months ago
Last Updated
2 weeks ago
Language
Python
License
NOASSERTION

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

chemistry-reproduce-the-scoring-path-before-you-replace-it

Use at implementation, experimentation and analysis when you are reproducing a published benchmark number and the source's scoring path is one you can read — which rows are scored, in what order, how many the loader drops, which epoch is reported, how tasks are pooled, over how many seeds. Covers implementing that path exactly before improving it, the one-row-per-step ladder from the published rule down to your own honest estimate, and why one un-replicated step makes the reproduction gap you report uninterpretable.

805 Updated 2 weeks ago
tangxiangru
AI & Automation Solid

add-the-baseline-back-on-the-split-you-cannot-score

Use at study design, implementation and every write thereafter, whenever the value you submit is assembled from parts - a fitted baseline plus a model's residual, a level plus a shape, a de-trended prediction that has to be re-trended, any inverse transform - and the validation arrays and the graded arrays are produced by separate calls. Covers assembling every split through one function, using the baseline you already fitted as a label-free reference vector on the graded split, why row count, header, dtype and finiteness cannot see this class of error, and putting the gate inside the writer rather than in a script somebody has to remember to run.

805 Updated 2 weeks ago
tangxiangru
AI & Automation Listed

prompt-and-grader-audit

Read an eval's prompt and its grader together as a single artefact and find where they contradict each other. Use when eval scores look wrong, a model is being punished for correct behaviour, scores are suspiciously low, or before trusting any eval result. Refuses to accept a score where the prompt and grader disagree.

0 Updated 1 weeks ago
ityaadiii