upgrade-a-live-service-safelylisted

Use when upgrading, rebuilding, or redeploying a service that is CURRENTLY SERVING production traffic — an LLM router, API gateway, web app, or any long-running daemon on a host you reach over SSH. Covers the build-in-place trap (build tooling that deletes the running service's working directory), CPU saturation starving both the service and sshd, detaching builds so an SSH pipe drop can't kill them mid-write, the release-dir + symlink swap pattern that makes rollback atomic, out-of-band health probing when you can no longer reach the box, and cloud-level reboot recovery. Also covers offloading the build to CI entirely (free arm64 GitHub runners), right-sizing a host by comparing runtime RSS against build peak, and smoke-testing a new release on a test port against a copy of the DB before swapping. Load this BEFORE running any build or install command on a host that is answering live requests.
TechNickAI/hermes-skills · ★ 0 · AI & Automation · score 76

Install: claude install-skill TechNickAI/hermes-skills

# Upgrade a Live Service Safely **Mission:** upgrade a service without taking it down — and when something does go wrong, recover fast with a rollback you prepared _before_ you needed it. 🔴 **Load this skill and its `scripts/` BEFORE writing any deploy tooling.** On One run spent ~40 minutes hand-rolling staging commands, skipped the test-port smoke entirely, and was about to cut over to production before anyone checked whether a packaged procedure already existed. It did: the bundled `scripts/stage_and_smoke.sh` already contained the test-port instance, the DB-copy isolation, the live-`MainPID` invariant, and the real-inference smoke — none of which the hand-rolled version had. **Reach for the packaged script before improvising one.** Adapting a proven script is faster than writing a worse one, and the parts you would have skipped are exactly the parts that catch problems. The governing rule: **never let the build and the running service share a directory, a CPU budget, or an SSH session.** Every serious outage in this class comes from violating one of those three. **The strongest version of that rule: don't build on the box at all.** Build in CI (free arm64 runners on public repos), ship an artifact, unpack to a new release dir, smoke it on a test port, then swap. Proven end-to-end — see `references/ci-offload-build-and-stage.md`, with a copy-and-adapt `templates/ci-standalone-build.yml` and `scripts/stage_and_smoke.sh`. Filling that template in for a specific service