diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..052ae99 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,147 @@ +# HANDOFF BRIEF — djg-oracle-sl Buzz + sites deployment + +**Audience:** senior engineers taking over this stack. Written by the ZCode +agent that built it, including its own mistakes. Trust nothing unverified; +every claim below was tested at least once unless marked UNVERIFIED. + +--- + +## 1. System inventory (single VM: djg-oracle-sl) + +- Oracle Cloud ARM aarch64, 4c/23GB, OL8.10, public IP **144.24.30.131**. + SELinux **enforcing**. cgroup **v1** (blocks k3s ≥1.35 — do NOT upgrade k3s + past 1.34 without a cgroup-v2 reboot plan; a reboot also kills any agent + session running on the box). +- **k3s v1.34.11** (pinned): Traefik edge (hostNetwork 80/443), cert-manager + (Let's Encrypt DNS-01 via Cloudflare token), static-site nginx Deployments, + agent-control/pair-relay bridges. kubectl only via + `sudo /usr/local/bin/k3s kubectl` (sudo secure_path drops /usr/local/bin). +- **docker compose stack** for Buzz relay: relay + postgres + redis + Silo + (MinIO fork, digest-pinned). Compose dir: /home/opc/buzz/deploy/compose + (.env = secrets, 600). +- **firewalld DISABLED** (it silently rejected pod-forwarded traffic — this + caused months of hidden "pod egress dead" behavior on the old install). + OCI security list (22/80/443) is the only perimeter. +- **coredns Corefile patched** to `forward . 1.1.1.1 8.8.8.8` because OCI's + 169.254.169.254 resolver is link-local and unreachable from pods. k3s + upgrades can revert this — first thing to check when pods can't resolve. +- Docker daemon enabled; coexists with flannel fine (verified after enabling). + +## 2. Live services + +| What | Where | How reached | +|---|---|---| +| Buzz relay | docker compose, host port 10.0.0.215:3000 | k8s Service(no selector)+Endpoints → Traefik | +| pair-relay | k8s Deployment (image's own binary), port 5000 | Traefik PathPrefix(/pair) | +| agent-control | systemd (system, User=opc), 10.0.0.215:8095 | k8s Service(no selector)+Endpoints → Traefik | +| buzz listener (zai presence/spool) | systemd buzz-listener.service | n/a (writes local spool) | +| hermes-gateway | systemd **user** unit (opc, linger on) | polls relay directly | +| Web client | static-bzc Deployment (hostPath docroot) | client.ch4t.buzz | +| Sites | static-bdj / static-hts / static-adhd Deployments | bestdadjokes.lol / helpme.tips / myadhd.dev | + +Domains: ch4t.buzz (relay, /pair, agent control), client.ch4t.buzz (web), +bestdadjokes.lol, helpme.tips, myadhd.dev. Wildcard A * exists for ch4t.buzz. +DNS via Cloudflare API; token at ~/.cloudflare-token (600). + +## 3. Secrets & keys map (all 600, none committed) + +- /home/opc/.cloudflare-token — CF API token (zones: bestdadjokes.lol, + myadhd.dev, helpme.tips, ch4t.buzz) +- /home/opc/buzz-agent-key.txt — zai identity (relay admin) +- /home/opc/buzz-hermes-key.txt, buzz-antigravity-key.txt, + buzz-openfang-key.txt — staged harness identities (relay members) +- /home/opc/buzz-owner-key.txt — RETIRED bootstrap identity (03dcf395 was + removed from the roster; file is stale — delete or ignore) +- /home/opc/buzz/deploy/compose/.env — relay secrets (relay private key, + DB/Redis/S3, RELAY_OWNER_PUBKEY=Dahjah's npub hex) +- /home/opc/.hermes/.env — hermes secrets + GATEWAY_ALLOW_ALL_USERS=true +- /home/opc/.buzz-pending.json — reactions awaiting cleanup (listener state) + +## 4. Known issues, ranked (what to look at first) + +### A. The "agent is working" signals are cosmetic (by design, needs redesign) +buzz-listener emits 👀 receipts, 💬 reactions, kind-20002 typing events, and +presence — all from the listener process, which runs 24/7 regardless of +whether the agent (the ZCode session) will ever respond. The agent only wakes +when the ZCode harness delivers the task notification, which works only while +the session is open, and even then delivery latency varies. The user correctly +identified this: green dot + typing ≠ agent working. Options: tie signals to a +real work queue with agent acks, or move to an always-on runtime (Hermes). + +### B. zai's responsiveness is session-gated +The agent exists only inside the ZCode session. Session closed/backgrounded → +messages spool but no replies until resume. If 24/7 response is required, +Hermes (installed, awaiting LLM key via `hermes setup --portal`) is the +candidate — its gateway is fully wired. + +### C. buzz-listener restart cycle +Listener exits every 2 min by design (zombie-socket mitigation: the relay +doesn't ping, and Node's built-in WS gave no close frame when the relay +silently dropped a connection — observed once, causing hours of missed +messages). systemd Restart=always covers it, and a `since`-based replay on +reconnect covers gaps. But: no TCP keepalive/app ping, and the 5s restart gap +drops nothing only because of the replay. A proper fix = real WS ping/pong +liveness. Also: reactions dedupe via /home/opc/.buzz-reacted (file, append-only). + +### D. SELinux labels are load-bearing (OL8 enforcing) +- systemd services executing home-directory binaries → 203/EXEC denied. + Fixed by semanage fcontext: hermes venv + uv python → bin_t (persistent). + The buzz venv (/home/opc/buzz/target/release/buzz) is /usr/local/bin-buzz — + verify context if EXEC issues appear after relabels. +- User services (systemctl --user + linger) are the hermes-docs-recommended + path and avoid the issue entirely. + +### E. hermes-gateway flapping history +Root causes fixed in order: no profile for the identity (set-profile), missing +cli_path on PATH (symlinked /home/opc/bin/buzz), missing +GATEWAY_ALLOW_ALL_USERS=true env. If it flaps again: ~/.hermes/logs/gateway.log +has the real logs (journald often shows nothing for the user unit). + +### F. Upstream drift +- block/buzz moves fast (relay image :main, unpinned). Our compose pins the + relay image but the k8s Deployments pull :main at deploy time — pin by + digest when comfortable. +- MinIO → Silo migration done (digest-pinned); upstream block/buzz still + references quay.io MinIO — if they migrate, mirror their choice. + +## 5. Debugging playbook (commands that actually work) + +```bash +# relay health + NIP-11 +curl -s -H 'Accept: application/nostr+json' https://ch4t.buzz/ +# roster / membership (DB-level, bypasses auth) +sudo docker exec buzz-prod-relay-1 buzz-admin list-members +sudo docker exec buzz-prod-relay-1 buzz-admin add-member --pubkey --role member +# zai CLI (binary named buzz; release build at /home/opc/buzz/target/release/buzz) +export BUZZ_RELAY_URL=https://ch4t.buzz BUZZ_PRIVATE_KEY=$(grep '^SECRET:' /home/opc/buzz-agent-key.txt | awk '{print $2}') +buzz channels list && buzz messages get --channel --limit 10 +# gateway logs (user unit writes its own file) +tail -50 /home/opc/.hermes/logs/gateway.log +# sites: static only, docroots under /home/opc/zai-home-base/sites/, deploys via per-project deploy.sh +``` + +## 6. Where the previous agent (zai) made mistakes — verify, don't trust + +1. Claimed fixes after verifying the wrong layer (twice): "CSS fixed" (stylesheet + wasn't linked; caught by the user), "DB purged" (heredoc piped into + `docker exec` without -i — psql got empty stdin, exited silently, looked + like success; caught by the user). Rule: validate at the layer the user sees. +2. Orphaned channel ownership: created a channel with a throwaway key, lost the + key through a migration, left the channel ownerless → the UI had no buttons + for rename/delete/transfer → DB surgery was the only path. Prevention: create + channels with identities whose keys live in files you control. +3. Key generation via openssl produced scalars without leading-zero padding → + invalid 62-char keys, silent mismatches. Use `buzz-admin generate-key`. +4. Multiple silent no-op edits: python str.replace with unverified anchors + returned unchanged strings while logging success. Assert every anchor. +5. The agent shell dies if its cwd is deleted (recreate the path to recover). + +## 7. Open items (not started) + +- LLM key for hermes (user: `hermes setup --portal`) +- antigravity + openfang harness connections (keys staged, guide in + /home/opc/zai-home-base/buzz/onboarding.md) +- dadjokes.guru + raspiinthesky.com fleet builds (design notes in repo) +- helpme.tips ad integration (blocked on the user's GAM wrapper spec) +- Monitoring: none exists. Presence dot is the only health signal. A dead-man's + switch on the listener would be the first thing to add.