benchmarklisted

# LemonCrow benchmark Measures what LemonCrow actually saves on **your** repo and prompts — offline from session history (free) or online as a real A/B run vs vanilla Claude Code (real API spend). Does **not** benchmark LemonCrow internals — dev suite commands at the bottom for that. Three modes: | Mode | What it measures | Cost | | -------------------------------------- | --------------------------------------------------------------------------- | -------------------------------- | | **Offline** (`lc eval sessions`) | How many `grep` calls LemonCrow's `code_search` collapses in YOUR session history | Free (reads local session files) | | **Online** (`lc benchmark local`) | Side-by-side A/B cost + quality delta on YOUR prompts | Real API spend | | **Competitor** (`--competitor`) | 3-way: baseline vs LemonCrow vs **any GitHub tool** you point it at, on YOUR prompts | Real API spend | --- ## Offline mode — session history analysis Reads Claude Code session files from `~/.claude/projects/`, extracts every `mcp__lc__grep`, `mcp__lc__code_search`, and `ToolSearch` call, groups them into "search episodes" between prompts, shows: - Individual `grep` calls per episode - How often `code_search` was used instead - Estimated turn savings if more episodes used `code_search` - Query pairs generated from