HeyBrian v4 RUNBOOK — recovery procedures

HeyBrian v4 RUNBOOK — recovery procedures

When something on the v4 stack is broken, find the symptom here and
follow the steps. Architecture context: heybrian_v4_architecture_260509.md.

Symptom: HeyBrian crashed → both ports cold → gateway can't reach Mac

The Lua dead-man's switch (phase 1.8) kills Lua ~15s after HeyBrian
dies. Both ports cold means the MacAccessGateway can't open a session
(it requires :8765). Chicken-and-egg.

Fix (autonomous, no Jonah needed):
1. /opt/agent/scripts/heybrian_relaunch.sh — bypasses the gateway via
the allowlist. pkill + open. HeyBrian back on :9100.
2. /opt/agent/scripts/hammerspoon_reload.sh — osascript hs.reload.
Lua back on :8765.

Both helpers are TEMPORARY (added 2026-05-09 for the overnight push).
Remove their entries from /root/.claude/hooks/brian-mac-access-gateway-guard.py
ALLOWED_PATTERN once the auto-recovery mechanism in HeyBrian (proposed:
HeyBrian self-fires hs.reload on first /v2 traffic when Lua is detected
dead) lands.

If the helpers don't work (Mac unreachable / SSH key changed):
1. Spotlight (Cmd+Space) → "HeyBrian" → Enter.
2. Hammerspoon menubar icon → Reload Config.

Symptom: Cpanel asks for token every visit

Token storage was switched from sessionStorage to localStorage in
commit 3761c7a. Survives tab close + browser quit. If the panel still
asks for a token, the browser's localStorage is empty for the origin.

Fix: open http://100.79.172.24:9100/cpanel#t=<token> once. JS
reads the hash, saves to localStorage, strips the hash. Next visit
goes straight to the panel.

Token: cat ~/.brian/cpanel_token on Mac.

Symptom: Cpanel served but session never shows in /cpanel/api/state

State machine ignores updates if ~/.heybrian/config.json has
native_overlays_enabled: false. The flag was meant to gate ONLY
the visual mount, but if the file is malformed or has the wrong value
the session updates may silently drop.

Fix: cat ~/.heybrian/config.json. If present and false, either
delete the file (default is true) or set true. Restart HeyBrian.

Symptom: HeyBrian crashes on /v2 lifecycle / overlay mount

Three known crashers, all fixed in commits eb7ef0c → state-machine push:

1. view.layer = parent while wantsLayer = true — over-release
   in the autorelease pool. Use view.wantsLayer = true then let parent = view.layer!. Don't replace.
2. Fade-out flash — capture parent.presentation()?.opacity,
   removeAllAnimations, set model to that value, then animate to 0.
3. Window teardown autorelease — NSWindow.close() with default
   releasedWhenClosed=true double-frees if also held in an array.
   Set isReleasedWhenClosed=false at mount; tear down via orderOut +
   ARC release (no .close()).

If a new overlay-related crash appears, get the .ips report from
~/Library/Logs/DiagnosticReports/HeyBrian-*.ips and look at the
faulting-thread frames around objc_release /
AutoreleasePoolPage::releaseUntil.

Symptom: Caddy restart loop, all *.jonahtebaa.com 000

Probable cause: a Caddyfile change that fails validation at startup.

Fix:
1. docker logs agent-caddy --tail 5 — find the error.
2. Edit /opt/agent/caddy/Caddyfile to fix.
3. NOTE: the /etc/caddy/Caddyfile bind mount can be stale. If the
container's view of the file (md5 inside container vs host) differs,
you need to docker restart agent-caddy, not just caddy reload.
4. If port :443 is stuck (after tailscale serve):
tailscale serve reset docker compose -f /opt/agent/caddy/docker-compose.yml up -d tailscale serve --bg --https=443 --set-path=/ http://127.0.0.1:6080
tailscaled binds the Tailscale IP at :443 with REUSEADDR, blocking
docker-proxy from binding 0.0.0.0:443 if it grabs first.

Symptom: Contract test harness fails / can't fetch HMAC secret

Harness fetches secret via gateway. If gateway is down (Lua dead),
fetch fails. Fix Lua first (hammerspoon_reload.sh), then re-run
harness. Cache file at /tmp/heybrian_hmac_secret.cache (TTL 600s)
will hold the secret in the meantime.

Symptom: Cpanel /api/state returns 401 even with token

CPanelAuth verifies via HMACGuard.constantTimeEquals against the
on-disk token. If the token was rotated (e.g., file rewritten) and
the running HeyBrian binary cached the old value at startup, restart
HeyBrian to reload.

Symptom: file.read or file.write returns "path denied"

Phase 1.12-A added a deny-list. The path resolved (~ expand, .. collapse,
symlink chase) lands under one of: ~/.ssh, ~/.gnupg, ~/.aws,
~/.azure, ~/.config/gcloud, ~/Library/Keychains, /Library/Keychains,
/private/etc, /etc, browser profiles (Chrome/Chromium/Firefox/Brave/
Safari), ~/Library/Cookies, ~/.brian, ~/.heybrian, ~/.hermes,
~/.brian_mac_secret.

This is intentional — Swift is intentionally stricter than Lua. If a
legitimate use case requires reading a denied path, edit
Sources/Executors.swift::denyPathPrefixes and ship a new build.

For one-off bypass via Lua during the dual-port shadow week, target the
Lua port (:8765) directly — Lua has no deny-list.

Symptom: audit log chain reports broken

Run:

curl -s http://100.79.172.24:9100/cpanel/api/audit/verify \
  -H "X-Brian-Cpanel-Token: $(cat ~/.brian/cpanel_token)"

{"intact": true} = chain healthy. {"intact": false, "reason": "line N: …"}
identifies the first broken link. Common causes:

1. Manual edit of ~/Library/Logs/Brian/control_center.jsonl — any
   change recomputes a different hash. Roll back the edit; the chain
   re-verifies on next call (read-only, no auto-repair).
2. Pre-1.12-B Lua-written entries — these have no hash field; the
   verifier skips them. They appear before any chained entry. Not a
   break.
3. Rotation seam — when the file hits 10 MB it rolls to a .timestamp
   suffix and the new file starts a fresh chain. The old file remains
   independently verifiable. NOT a break.

If genuine tampering is suspected, copy the JSONL off-host (Hetzner via
mac_ssh / Tailscale) and diff against any external mirror.

Modes: the endpoint defaults to loose (tolerates Lua entries
during the dual-port shadow week). Append ?strict=true post-1.11
cutover for the strong guarantee that catches append-only and
trailing-replace attacks gemini+codex flagged.

Symptom: TCC permission denials in /v2/action results

Phase 1.10 added explicit pre-flight checks. Caps that return
accessibility permission denied / screen recording permission denied
need Jonah to grant in System Settings. Use the cpanel "Warm up
permissions" button (POST /cpanel/api/permissions/warmup) to surface
all dialogs at once.

full_disk_access shows DENIED currently — HeyBrian.app needs to be
added to System Settings → Privacy & Security → Full Disk Access for
file.read on ~/Library/Messages, ~/Library/Mail, etc.

Symptom: Mac unreachable from Hetzner (Tailscale 000)

Mac sleep + wake-on-LAN limits. Tailscale presence pings every minute
or so. If Mac is asleep, expect 30-60s warmup after movement.

/opt/agent/scripts/heybrian_mac_watchdog.py cron alerts LOGS after
3min of unreachability — that's expected during normal sleep cycles.
Alert is level=low only (never SIP) per gemini's no-false-positive-
ring rule from 1.1c round-table.

When in doubt — fall back paths

• Direct HMAC-signed call to HeyBrian (no gateway): cached secret at
  /tmp/heybrian_hmac_secret.cache + openssl dgst -sha256 -hmac +
  curl to :9100/v2/action. Same pattern the contract harness uses.
• Direct browser to http://100.79.172.24:9100/cpanel over Tailscale —
  no Caddy, no gate auth, just the token.
• Hammerspoon Console — osascript -e 'tell application "Hammerspoon" to execute lua code "..."' — works as long as Hammerspoon itself
  is running (the v3 module's HTTP server may be down via DMS).