← ClaudeAtlas

promptfoo-evaluationlisted

Use for any question or task about Promptfoo or adding quick regression checks for prompts or RAG pipelines in CI, including turning written requirements into assertions: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.
testland/qa-skills · ★ 0 · AI & Automation · score 72
Install: claude install-skill testland/qa-skills
# promptfoo-evaluation [pf-intro]: https://promptfoo.dev/docs/intro/ [pf-config]: https://promptfoo.dev/docs/configuration/guide/ A `promptfooconfig.yaml` declares **providers** (LLMs under test), **prompts** (templates with `{{var}}` placeholders), **tests** (input variables + assertions); `promptfoo eval` runs the cross-product (per [pf-config][pf-config]). ## When to use - The repo has a `promptfooconfig.yaml` or the user wants to author one. - The user needs prompt regression suites - same inputs, multiple providers, fail-on-diff. - A CI workflow needs an eval gate on prompt or model changes. - The team prefers vendor-neutral eval (vs OpenAI Evals' OpenAI-first posture). ## Step 1 - Install Per [github.com/promptfoo/promptfoo][pf-gh]: [pf-gh]: https://github.com/promptfoo/promptfoo ```bash npm install -g promptfoo # or brew install promptfoo # or one-off npx promptfoo@latest ``` Per [pf-gh][pf-gh]: "Node.js 20.20+ or 22.22+ is required for npm and npx usage." ## Step 2 - First eval Initialize from a built-in example, then run: ```bash promptfoo init --example getting-started cd getting-started promptfoo eval promptfoo view # opens HTML report ``` (Commands per [pf-gh][pf-gh].) A minimal `promptfooconfig.yaml` per [pf-config][pf-config]: ```yaml prompts: - file://prompt1.txt providers: - openai:gpt-5-mini - anthropic:claude-haiku-4-5 tests: - vars: language: French input: Hello world assert: - type: contains-json ``` P