math-equal-effort-baselines-and-knob-sweeps

Solid

Use at study design when the source names competing algorithms and they are about to become a related-work paragraph instead of arms. Covers running every named baseline at equal tuning effort, and sweeping the parameter you claim credit for.

AI & Automation 804 stars 25 forks Updated today NOASSERTION

Install

View on GitHub

Quality Score: 82/100

Stars 20%
97
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Run every named baseline as a real arm at equal effort, and sweep the knob you claim credit for An algorithmic claim is a comparison, so the baseline suite is a build item, not a related-work paragraph. Enumerate the named competitors from the references shipped with the problem, implement or install each as a real arm, and run them on identical instances at an identical budget, reported in the same table and the same figure. Give every arm the same tuning effort and the same optional machinery: if a baseline gets restarts, warm starts, preconditioning or a tuned step size, your method gets them too, and the reverse. An asymmetric enhancement is the commonest way an apparent speed-up turns out to be a configuration difference. Attribute the gain. Ablate the novel component on and off, then sweep that component's own hyper-parameter across its range to locate the optimum and the point where gains saturate or reverse. A component reported only as present or absent leaves a reader unable to distinguish a mechanism from a lucky setting. Report quality and cost as a pair for the same comparison -- objective value with iterations or time, solution quality with memory or node expansions, accuracy with throughput -- and give the break-even factor. Break results out per instance family and per problem regime the benchmark distinguishes, including families outside your method's design or training regime and families that are degenerate at the shipped settings. Each still gets its ...

Details

Author
tangxiangru
Repository
tangxiangru/AutoR
Created
5 months ago
Last Updated
today
Language
Python
License
NOASSERTION

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

life-benchmark-against-the-incumbent

Use at study design when a life-science method result is about to be reported on its own numbers. Covers the head-to-head against the incumbent tool, the cost table that goes with it, and finding an orthogonal truth set the method was not fitted to.

804 Updated today
tangxiangru
AI & Automation Solid

chemistry-accuracy-and-cost-for-every-module-you-swap-in

Use at study design, through experimentation and again at analysis when the method under test is a drop-in replacement for a standard layer — a different basis, kernel, activation family or transform — and the source claims the replacement is both more accurate and cheaper. Covers giving every alternative module a cell in the accuracy column and in the cost column, fixing one matching convention across both, and dividing the runtime by the invariant already sitting in your own results file before you publish a contradiction of the source's ratio.

804 Updated today
tangxiangru
AI & Automation Featured

baseline-comparison-audit

Audit whether a paper's baseline comparisons are COMPLETE, FAIR, and SIGNIFICANT: a required recent SOTA baseline is missing while 'best/SOTA' is claimed (HP-MISSING-BASELINE); a baseline is undertuned / given less compute-tuning-data, run at a mismatched config, or the equal-budget ablation-as-baseline is absent (HP-WEAK-BASELINE); 'outperforms' is asserted over overlapping error bars or with no variance/seeds (HP-SIG-OVERLAP); and a cross-row 'improves over baseline by X%' is arithmetically wrong (HP-DELTA-ERROR, cross-row form only). A versioned per-domain baseline profile + a live leaderboard/recency search are assembled by the EXECUTOR as structured facts; a fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per dimension) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Works at L0 (stated comparisons) and deepens at L2 (configs/result files). A completeness question it cannot settle internally becomes needs_external_check,

139 Updated 3 days ago
wanshuiyin