posthog-experiment-readerlisted

Read a PostHog A/B test or feature flag split honestly at indie-hacker volumes - "can I call it?", "did the variant win?", "should I ship this paywall / pricing / onboarding change?", "is the new version better than the old one?" - by checking the setup (exposures per variant, sample ratio mismatch, people in both arms), pulling per-variant exposures and conversions, computing absolute and relative lift with its margin of error, flagging peeking, short runs and mix-shift, reading replays per variant, and returning a verdict (ship, wait until N, redesign, investigate) instead of a false positive. Also use it for informal before/after comparisons of two versions when the user asks whether a change "worked". Not for designing a new experiment from scratch.
josephbedminster/indie-marketing-skills · ★ 1 · AI & Automation · score 74

Install: claude install-skill josephbedminster/indie-marketing-skills

# Reading an experiment honestly Indie experiments die from small numbers, not from bad ideas. 300 people per variant can show "+49 %" that is pure noise. The job is to say which of these is true: the variant wins, it loses, it makes no measurable difference, or it's too early and here is when it won't be. Read `.claude/posthog-funnel-map.md` first (run `posthog-funnel-map` if it doesn't exist) for the conversion events, noise filter and business baseline. Tool conventions are the same as in that skill. Everything here is read-only: never stop, ship or edit an experiment or flag without the user's explicit yes. **Project conventions** in the map (report language and style, report location, spend sources, extra context sources) override this skill's defaults. ## 1. Resolve the experiment - A PostHog experiment: `experiment-get-all` (or `system.experiments`), then `experiment-get`. Note the flag key, variants and their split, overall rollout, start date, status, exposure event, test-account filtering, and stats method (Bayesian by default, or frequentist). - A plain feature flag used as a test: `feature-flag-get-all`, then the flag's variants and rollout. The exposure is `$feature_flag_called` with `$feature_flag = '<key>'`. - An informal "before vs after" (no randomization): say upfront that it is not an experiment, then compare two windows of equal length and the same weekdays with the investigation skill's recipe C, and treat any difference as a hypothes