benchmark-runner

Featured

Designs structured benchmarks comparing algorithms, models, or implementations with metrics, test cases, hardware context, and reproduction steps. Triggers on: "benchmark", "compare performance", "which is faster", "latency comparison", "run benchmark", "throughput test", "speed test".

Testing & QA 313 stars 46 forks Updated 4 days ago MIT

Install

View on GitHub

Quality Score: 93/100

Stars 20%
83
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Benchmark Runner Standardizes performance comparison methodology: metric selection, test case design, environment capture, result formatting, and tradeoff analysis. Produces reproducible benchmark reports that support informed decisions — not just "A is faster than B" but "A is faster for small inputs while B scales better." ## Reference Files | File | Contents | Load When | | ----------------------------------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------- | | `references/metric-selection.md` | Metric catalog (latency percentiles, throughput, memory, accuracy), selection criteria per task type | Always | | `references/test-case-design.md` | Representative input selection, scale variation, edge case coverage, warmup strategies | Always | | `references/environment-capture.md` | Hardware/software context recording, reproducibility requirements, variance control | Always | | `references/statistical-rigor.md` | Sample sizing, variance measurement, significance testing, outlier handling | Results need statistical validation | ## Prerequisites - Clear candidates to compare (at least 2) -...

Details

Author
Mathews-Tom
Repository
Mathews-Tom/armory
Created
5 months ago
Last Updated
4 days ago
Language
Python
License
MIT

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category