← ClaudeAtlas

llm-regression-suite-authorlisted

Use for any question or task about catching LLM quality regressions, including checking a new model version or provider against the old one and prompt tweaks that break other cases. Builds a versioned golden dataset, deterministic runs, exact, semantic, judge and rubric checks, and a pass-rate CI gate with a baseline diff across model versions.
testland/qa-skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill testland/qa-skills
# llm-regression-suite-author [pf-config]: https://promptfoo.dev/docs/configuration/guide/ [pf-testcases]: https://promptfoo.dev/docs/configuration/test-cases/ [pf-asserts]: https://promptfoo.dev/docs/configuration/expected-outputs/ [pf-ref]: https://promptfoo.dev/docs/configuration/reference/ [pf-cli]: https://promptfoo.dev/docs/usage/command-line/ [pf-openai]: https://promptfoo.dev/docs/providers/openai/ [pf-gha]: https://promptfoo.dev/docs/integrations/github-action/ [de-gh]: https://github.com/confident-ai/deepeval [de-start]: https://deepeval.com/docs/getting-started [de-datasets]: https://deepeval.com/docs/evaluation-datasets A regression suite for LLM outputs pairs a **versioned golden dataset** (stable inputs + expected outputs) with **deterministic eval settings** and a **pass-rate gate** so that promoting a model from v1 to v2 surfaces quality drops before they reach production. The per-tool skills (`promptfoo-evaluation`, `deepeval-evaluation`) handle tool mechanics; this skill handles the versioning workflow that neither covers. ## When to use - A vendor announces a model update (e.g., `gpt-4.1` to `gpt-5`) and you need a before/after quality comparison on your own inputs. - A prompt regression suite exists but has no stable dataset version tied to each model snapshot, so regressions are discovered in production. - CI must gate a model-version bump the same way it gates a code change. ## Step 1 - Structure the golden dataset A golden dataset is a version