task

Solid

Turn a natural-language description into one or more coder-eval task YAML files — minimal prompts, weighted success criteria that check output content, validated with `coder-eval plan`. Use when the user wants to write, add, or generate an evaluation task.

AI & Automation 116 stars 2 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 80/100

Stars 20%
69
Recency 20%
100
Frontmatter 20%
40
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Author a coder-eval task You are writing coder-eval task YAML. The user's request is: `$ARGUMENTS` If `$ARGUMENTS` is empty, ask what the task should test. Do not invent a subject. Good tasks use simple prompts: state the goal and the expected output, then let the agent work out the approach. A single request can produce **several** task files — "create tasks for all the registry subcommands" means one task per subcommand. ## Step 1 — Understand the request, and check the CLI is there Run `coder-eval --version` first. Steps 6 and 7 both shell out to it, and finding that out *after* writing several task files means the user gets a bare `command not found` with nothing to act on. Installing this plugin did not install the CLI. If it is missing, follow `${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md`: offer the install, **ask before running it**, and confirm with `coder-eval --version` afterwards. Never install unprompted, and do not write any task files if the user declines. That reference also covers the other half of the version check — whether this project pins a coder-eval version, and what to do when the installed one does not match it. Then establish: - **What is being tested** — which tool, SDK, CLI, skill, or capability? - **How many tasks** — one operation, or several? - **Difficulty** — smoke, basic, or intermediate? - **Dependencies** — network, packages, starter files, external services? State any assumptions you make rather than silently picking. ## Step...

Details

Author
UiPath
Repository
UiPath/coder_eval
Created
1 months ago
Last Updated
today
Language
Python
License
Apache-2.0

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category