← ClaudeAtlas

learn-an-interfacelisted

Drive a real graphical interface from an agent — a window, a ribbon, a dialog, a canvas — reaching it through the accessibility tree where one exists and learning its coordinates where none does, instead of hard-coding them. Covers the order of preference (API, CLI, file format, library, UI Automation / AX / AT-SPI, macro, and only then pixels), the input calls that actually reach a modern application, how to measure a window so clicks land, a small bandit that finds the unknown detail and remembers it, and the measured traps that make a working automator report success while doing nothing. Ships a runnable learning loop. Use when a task needs a program driven and it has no API, no command-line tool, no scriptable interface and no readable file format; when choosing between UI Automation and screenshot-and-click; when clicks land on the wrong thing or nowhere; when a keyboard or screenshot tool appears to do nothing; when a GUI step must be repeatable without a human hand.
ubik-dsh/deepseek-harness-skills · ★ 0 · AI & Automation · score 63
Install: claude install-skill ubik-dsh/deepseek-harness-skills
# Learning an interface Sometimes the only handle on a thing is a picture of a window. This skill is for that case: how to decide whether to build it, how to see and touch a real window, and how to **learn** the details nobody documented instead of writing coordinates into a script and watching them go stale. Read it in order. Step 1 is a refusal, and most of the value is there. --- ## Step 1 — Try to not do this Driving a GUI is the last resort. The question is not "can this be automated?" — it always can — but "is there a supported way in?" Stop at the first yes: | ask | if yes | |---|---| | an API, SDK or service? | use it | | a command-line tool? | use it | | is it a file format you can read and write? | do that | | a library for it? | use that | | **does it expose an automation tree — UI Automation, AX, AT-SPI?** | **use it** | | can the application script itself — macro, plugin, config, URL scheme? | use that | | **nothing, and the only way in is pixels and clicks** | continue | **The automation tree is the rung this skill was first written without, and a survey of what already exists on GitHub is what exposed the gap.** It goes above pixels because it is a documented, platform-versioned interface, and above macros because it is uniform across applications. Measured on Paint: ``` Pencil automationId='PencilTool' supports Toggle Off -> On, no cursor Colors Black, Gray, Dark red, Red, Orange, Yellow, Green, Turquoise, Indigo Panes only ScrollViewer - th