experiment-loop

Solid

Autonomous experiment loop: hypothesize > modify > test > evaluate > keep/discard > repeat. Run N experiments automatically with measurable metrics. Works for performance optimization, A/B testing, prompt engineering, and any measurable improvement task.

AI & Automation 552 stars 45 forks Updated 6 days ago MIT

Install

View on GitHub

Quality Score: 91/100

Stars 20%
91
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Experiment Loop Autonomous, iterative improvement inspired by Karpathy's autoresearch methodology. Define a metric, set a target, and let the loop run until the target is met or the iteration limit is reached. ## The 5-Step Loop ``` 1. HYPOTHESIZE -> Form a specific, falsifiable improvement hypothesis 2. MODIFY -> Apply the minimal code/config/prompt change 3. TEST -> Run the measurement suite (benchmarks, tests, evals) 4. EVALUATE -> Compare result against baseline and previous best 5. DECIDE -> KEEP if better, DISCARD (git stash pop --index) if worse | Repeat until target met OR max_iterations reached ``` Each iteration is atomic: one hypothesis, one change, one measurement, one decision. ## Experiment Definition Define an experiment in your task or in `thoughts/EXPERIMENTS.md`: ```yaml experiment: name: "reduce-api-latency" metric: "p95 response time (ms)" baseline: 340 target: 200 direction: minimize # minimize | maximize max_iterations: 10 # hard cap, never exceed measurement_cmd: "npm run bench:api" measurement_key: "p95" # JSON key from bench output scope: "src/api/" # files the loop is allowed to touch ``` ### Key Fields | Field | Description | |-------|-------------| | `metric` | Human-readable name of what you are measuring | | `baseline` | Measured value before any changes (run this first) | | `target` | Success condition -- loop exits when this is met | | `direction`...

Details

Author
vibeeval
Repository
vibeeval/vibecosystem
Created
8 months ago
Last Updated
6 days ago
Language
C#
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category