creating-a-coral-task

Featured

Author a new CORAL task — the three pieces that must line up (`task.yaml`, `seed/`, a packaged `grader/`), the `coral init` → `coral validate` → smoke-test loop, and how to pick a grader pattern (stdout float, test pass-rate, ratio-vs-baseline, multi-metric, or an LLM rubric judge). Use whenever the user wants to create a CORAL task, write or wire a grader, port a benchmark into CORAL, score open-ended outputs (reports/memos) with a judge, or debug a grader that crashes on the seed / ranks the leaderboard backwards / leaks the answer key. Deep references for the TaskGrader API, grader patterns, rubric judges, and the full task.yaml schema live alongside this skill.

AI & Automation 910 stars 120 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 93/100

Stars 20%
99
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Creating a CORAL task A CORAL task is **three things that must line up**. Scaffold them with `coral init`, then iterate `edit → coral validate` until the grader scores the seed. ``` my-task/ ├── task.yaml # config: name, description, grader entrypoint, agent count ├── seed/ # starter code agents see at t=0 (this is workspace.repo_path) │ └── solution.py └── grader/ # standalone Python package — gets its own isolated venv ├── pyproject.toml └── src/my_task_grader/ ├── __init__.py └── grader.py # class Grader(TaskGrader): ... ``` The packaged grader is the **only supported form** — it gives the grader an isolated venv and bundles everything the eval needs (grader code, helpers, hidden answer keys). There is no `eval/grader.py` auto-discovery anymore. > **Optimizing code the user already has?** Scaffold inside a `.coral_workspace/` at the root of their project (gitignored), and copy the code to optimize into `seed/` — keeps CORAL's task/results out of their source tree. The `coral-quickstart` skill has the end-to-end `.coral_workspace/` flow; this skill covers the grader you'll write once the code is in `seed/`. > > **"Optimize this" is a build instruction, not a question — never answer it with a process menu.** A 1/2/3 like "point me to a task / create one / optimize outside coral" is the failure mode; do not produce it. The absence of a `task.yaml` is not ambiguity — it just means you build one from the current repo. Con...

Details

Author
Human-Agent-Society
Repository
Human-Agent-Society/CORAL
Created
5 months ago
Last Updated
today
Language
Python
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Featured

coral-new-task

End-to-end recipe for adding a new task under `examples/` — the three pieces that have to line up (`task.yaml`, `seed/`, and `grader/`), what to put in each, the `TaskGrader` API surface, the `coral validate` → smoke-test loop, and the common mistakes (repo_path pointing at the wrong dir, score direction backwards, hidden answer keys leaking into seed/, grader writing to codebase_path which the daemon force-removes, private-vs-public confusion, missing `run()` signature). Use whenever the user wants to add a new CORAL task or port an existing benchmark into CORAL.

910 Updated today
Human-Agent-Society
AI & Automation Featured

coral-quickstart

The fast path from zero to a running CORAL experiment — what CORAL is and when to reach for it, installing the `coral` CLI, registering a runtime with `coral setup`, and the `.coral_workspace/` convention for pointing CORAL at code you already have and want optimized. Use this whenever the user asks "what is coral", "should I use coral for this", wants to install or get coral set up, hits a "command not found" for coral or doesn't have it installed yet, or says "use coral to optimize / speed up / improve this code" and you need the end-to-end onboarding from install to a launched run. Hands off to `setting-up-coral` (runtime bindings), `creating-a-coral-task` (grader authoring), and `running-coral-experiments` (operating a run) for depth.

910 Updated today
Human-Agent-Society
AI & Automation Featured

coral-extend

Add a new component to the CORAL framework itself — a new agent runtime under `coral/agent/builtin/` (claude_code/codex/cursor_agent style), a new CLI command in `coral/cli/`, a new bundled skill or subagent template under `coral/template/skills/` or `coral/template/agents/`, a new hook in `coral/hooks/`, a new field in `coral/config.py`, or a framework-level extension to the grader stack under `coral/grader/`. NOT for writing a per-task grader or adding an example task — use `coral-new-task` for that. NOT for debugging existing code — use `coral-debug`.

910 Updated today
Human-Agent-Society