skill-evaluator

Solid

Evaluates agent skills against Anthropic's best practices. Use when asked to review, evaluate, assess, or audit a skill for quality. Analyzes SKILL.md structure, naming conventions, description quality, content organization, and identifies anti-patterns. Produces actionable improvement recommendations.

AI & Automation 422 stars 31 forks Updated 3 weeks ago MIT

Install

View on GitHub

Quality Score: 91/100

Stars 20%
87
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Skill Evaluator (WIP) Evaluates skills against Anthropic's official best practices for agent skill authoring. Produces structured evaluation reports with scores and actionable recommendations. ## Quick Start 1. Read the skill's SKILL.md and understand its purpose 2. Run automated validation: `scripts/validate_skill.py <skill-path>` 3. Perform manual evaluation against criteria below 4. Generate evaluation report with scores and recommendations ## Evaluation Workflow ### Step 1: Automated Validation Run the validation script first: ```bash scripts/validate_skill.py <path/to/skill> ``` This checks: - SKILL.md exists with valid YAML frontmatter - Name follows conventions (lowercase, hyphens, max 64 chars) - Description is present and under 1024 chars - Body is under 500 lines - File references are one-level deep ### Step 2: Manual Evaluation Evaluate each dimension and assign a score (1-5): #### A. Naming (Weight: 10%) | Score | Criteria | |-------|----------| | 5 | Gerund form (-ing), clear purpose, memorable | | 4 | Descriptive, follows conventions | | 3 | Acceptable but could be clearer | | 2 | Vague or misleading | | 1 | Violates naming rules | **Rules**: Max 64 chars, lowercase + numbers + hyphens only, no reserved words (anthropic, claude), no XML tags. **Good**: `processing-pdfs`, `analyzing-spreadsheets`, `building-dashboards` **Bad**: `pdf`, `my-skill`, `ClaudeHelper`, `anthropic-tools` #### B. Description (Weight: 20%) | Score | Criteria | |-------|--...

Details

Author
gotalab
Repository
gotalab/skillport
Created
11 months ago
Last Updated
3 weeks ago
Language
Python
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

skill-evaluation

Evaluate any agent skill against a merged framework — Anthropic's Claude Code best practices plus Matt Pocock's writing-great-skills methodology — across 4 axes (Trigger, Structure, Steering, Pruning). Produces an evidence-cited scorecard (0–100), a weighted overall score, and diagnosed failure modes with prioritized fixes. Use when the user asks to evaluate, rate, or audit a skill ("evaluate this skill", "skill scorecard", "review SKILL.md"), or to compare two skills.

138 Updated 3 days ago
fabricioctelles
Code & Development Listed

skill-eval-harness

Write, evaluate and maintain agent skills with measured evidence rather than assertion. Covers SKILL.md structure and frontmatter, writing a description that triggers reliably without firing on neighbouring topics, progressive disclosure through references and scripts, and building eval suites across three families - usefulness with skill on versus off, discovery, and adversarial discovery. Use when creating a new skill, reviewing or debugging one that does not trigger or does not help, deciding whether a skill earns its context cost, writing an eval suite or grader, or interpreting skill eval results. Also use when asked whether a skill is any good.

0 Updated 1 weeks ago
Premchand006
AI & Automation Listed

skill-evaluator

Rigorously evaluate an Agent Skill end-to-end across ANY coding-agent CLI — verify its scripts emit the documented numbers (deterministic checks), test whether its description triggers on the right prompts, and measure whether an agent following the SKILL.md beats a no-skill baseline (with/without pass-rate delta, mean ± stddev, benchmarked). Use whenever you need to test, benchmark, validate, grade, or quantify a skill's quality, check if a skill "actually works," compare two skill versions, optimize a skill's triggering, or set up an eval suite — even if the user just says "is this skill any good," "does my skill work," or "benchmark this skill." Drives Claude Code, OpenAI Codex, Antigravity (agy), Cursor, GitHub Copilot, Amp, opencode, or Grok in headless mode.

94 Updated 2 days ago
HeshamFS