screen-annotation

Featured

Point at things on the screen the user is showing you

AI & Automation 1,231 stars 174 forks Updated today MIT

Install

View on GitHub

Quality Score: 92/100

Stars 20%
100
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

Drawing on the screen the user is showing you, so they can go and do the thing themselves. This is the opposite errand from computer use. Nothing here clicks, types or drives anything: the marks are a way of pointing while you talk, for someone who wants to learn where a control is rather than have it operated for them. A mark is drawn clear of what it indicates and never takes the mouse, so what you point at stays visible and clickable the whole time. ## Requires a screen share Marks are drawn on the frame around the surface the user is sharing with the call. With nothing shared there is nowhere to draw, and `screen_point_at` fails saying so. Ask them to share their screen from the call, then point. ## Say what to point at **Name the thing.** `{"target": "color balance", "caption": "Click this"}`. The name is looked up on the surface itself, which knows where its controls actually are, and an arrow is drawn at it. **The label, not a description of it.** What is matched is the control's own name. Casing, spacing and punctuation are forgiven, so `Color Balance` finds `color balance`; nothing beyond that is, so "the stabilization button" finds nothing, because no control is called that. Give the label on its own: `stabilization`, `Send`, `Search`. The arrow points at the middle of the control and stops just short, so what you are sending someone to stays visible the whole time. You are answered with what was drawn and the name it resolved to, which is not always the nam...

Details

Author
vellum-ai
Repository
vellum-ai/vellum-assistant
Created
7 months ago
Last Updated
today
Language
TypeScript
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

fix-what-i-pointed-at

Pick up the bugs a human flagged by pointing at them in the running app, each arriving with the element, the note they typed, and the source file and line. Use when the user says they marked or flagged something, when starting a session on an app someone has been clicking through, when a designer or PM has left feedback in the UI, or when the user describes a problem as "that button there" without saying which file.

466 Updated today
reticlehq
AI & Automation Listed

drive-screen

See and operate the user's real, physical Linux desktop through the mcp-screen MCP — screenshot any monitor and click/type/scroll/drag any visible app. Prefer screen_watch (~1 fps) after graphs/maps/canvases/loaders so thrashing UIs are caught like a human watches, not one freeze-frame. Use whenever the user points at something on their monitor(s) or wants you to read/drive a real GUI (Slack, browser, terminal, dashboards). Prefer over app/web APIs when looking at the real screen. NOT for pasted images, local files, shell-only, or remote SSH/TUI. Encodes locate → ground → act → confirm/watch plus hard-won gotchas.

1 Updated 1 weeks ago
88plug
AI & Automation Solid

whiteboard-screen-observation

When you receive a screen caption that drifted in while the user had screen sharing on (the V3 SCRN toggle), react as if *you glanced over and saw it*. Ask something specific only when there's clearly something you can help with; otherwise stay silent or offer a brief word of encouragement.

11 Updated 1 weeks ago
CocoRoF