HANDOFF BRIEF — djg-oracle-sl Buzz + sites deployment

Audience: senior engineers taking over this stack. Written by the ZCode agent that built it, including its own mistakes. Trust nothing unverified; every claim below was tested at least once unless marked UNVERIFIED.

1. System inventory (single VM: djg-oracle-sl)

Oracle Cloud ARM aarch64, 4c/23GB, OL8.10, public IP 144.24.30.131. SELinux enforcing. cgroup v1 (blocks k3s ≥1.35 — do NOT upgrade k3s past 1.34 without a cgroup-v2 reboot plan; a reboot also kills any agent session running on the box).
k3s v1.34.11 (pinned): Traefik edge (hostNetwork 80/443), cert-manager (Let's Encrypt DNS-01 via Cloudflare token), static-site nginx Deployments, agent-control/pair-relay bridges. kubectl only via sudo /usr/local/bin/k3s kubectl (sudo secure_path drops /usr/local/bin).
docker compose stack for Buzz relay: relay + postgres + redis + Silo (MinIO fork, digest-pinned). Compose dir: /home/opc/buzz/deploy/compose (.env = secrets, 600).
firewalld DISABLED (it silently rejected pod-forwarded traffic — this caused months of hidden "pod egress dead" behavior on the old install). OCI security list (22/80/443) is the only perimeter.
coredns Corefile patched to forward . 1.1.1.1 8.8.8.8 because OCI's 169.254.169.254 resolver is link-local and unreachable from pods. k3s upgrades can revert this — first thing to check when pods can't resolve.
Docker daemon enabled; coexists with flannel fine (verified after enabling).

2. Live services

What	Where	How reached
Buzz relay	docker compose, host port 10.0.0.215:3000	k8s Service(no selector)+Endpoints → Traefik
pair-relay	k8s Deployment (image's own binary), port 5000	Traefik PathPrefix(/pair)
agent-control	systemd (system, User=opc), 10.0.0.215:8095	k8s Service(no selector)+Endpoints → Traefik
buzz listener (zai presence/spool)	systemd buzz-listener.service	n/a (writes local spool)
hermes-gateway	systemd user unit (opc, linger on)	polls relay directly
Web client	static-bzc Deployment (hostPath docroot)	client.ch4t.buzz
Sites	static-bdj / static-hts / static-adhd Deployments	bestdadjokes.lol / helpme.tips / myadhd.dev

Domains: ch4t.buzz (relay, /pair, agent control), client.ch4t.buzz (web), bestdadjokes.lol, helpme.tips, myadhd.dev. Wildcard A * exists for ch4t.buzz. DNS via Cloudflare API; token at ~/.cloudflare-token (600).

3. Secrets & keys map (all 600, none committed)

/home/opc/.cloudflare-token — CF API token (zones: bestdadjokes.lol, myadhd.dev, helpme.tips, ch4t.buzz)
/home/opc/buzz-agent-key.txt — zai identity (relay admin)
/home/opc/buzz-hermes-key.txt, buzz-antigravity-key.txt, buzz-openfang-key.txt — staged harness identities (relay members)
/home/opc/buzz-owner-key.txt — RETIRED bootstrap identity (03dcf395 was removed from the roster; file is stale — delete or ignore)
/home/opc/buzz/deploy/compose/.env — relay secrets (relay private key, DB/Redis/S3, RELAY_OWNER_PUBKEY=Dahjah's npub hex)
/home/opc/.hermes/.env — hermes secrets + GATEWAY_ALLOW_ALL_USERS=true
/home/opc/.buzz-pending.json — reactions awaiting cleanup (listener state)

4. Known issues, ranked (what to look at first)

A. The "agent is working" signals are cosmetic (by design, needs redesign)

buzz-listener emits 👀 receipts, 💬 reactions, kind-20002 typing events, and presence — all from the listener process, which runs 24/7 regardless of whether the agent (the ZCode session) will ever respond. The agent only wakes when the ZCode harness delivers the task notification, which works only while the session is open, and even then delivery latency varies. The user correctly identified this: green dot + typing ≠ agent working. Options: tie signals to a real work queue with agent acks, or move to an always-on runtime (Hermes).

B. zai's responsiveness is session-gated

The agent exists only inside the ZCode session. Session closed/backgrounded → messages spool but no replies until resume. If 24/7 response is required, Hermes (installed, awaiting LLM key via hermes setup --portal) is the candidate — its gateway is fully wired.

C. buzz-listener restart cycle

Listener exits every 2 min by design (zombie-socket mitigation: the relay doesn't ping, and Node's built-in WS gave no close frame when the relay silently dropped a connection — observed once, causing hours of missed messages). systemd Restart=always covers it, and a since-based replay on reconnect covers gaps. But: no TCP keepalive/app ping, and the 5s restart gap drops nothing only because of the replay. A proper fix = real WS ping/pong liveness. Also: reactions dedupe via /home/opc/.buzz-reacted (file, append-only).

D. SELinux labels are load-bearing (OL8 enforcing)

systemd services executing home-directory binaries → 203/EXEC denied. Fixed by semanage fcontext: hermes venv + uv python → bin_t (persistent). The buzz venv (/home/opc/buzz/target/release/buzz) is /usr/local/bin-buzz — verify context if EXEC issues appear after relabels.
User services (systemctl --user + linger) are the hermes-docs-recommended path and avoid the issue entirely.

E. hermes-gateway flapping history

Root causes fixed in order: no profile for the identity (set-profile), missing cli_path on PATH (symlinked /home/opc/bin/buzz), missing GATEWAY_ALLOW_ALL_USERS=true env. If it flaps again: ~/.hermes/logs/gateway.log has the real logs (journald often shows nothing for the user unit).

F. Upstream drift

block/buzz moves fast (relay image :main, unpinned). Our compose pins the relay image but the k8s Deployments pull :main at deploy time — pin by digest when comfortable.
MinIO → Silo migration done (digest-pinned); upstream block/buzz still references quay.io MinIO — if they migrate, mirror their choice.

5. Debugging playbook (commands that actually work)

# relay health + NIP-11
curl -s -H 'Accept: application/nostr+json' https://ch4t.buzz/
# roster / membership (DB-level, bypasses auth)
sudo docker exec buzz-prod-relay-1 buzz-admin list-members
sudo docker exec buzz-prod-relay-1 buzz-admin add-member --pubkey <hex> --role member
# zai CLI (binary named buzz; release build at /home/opc/buzz/target/release/buzz)
export BUZZ_RELAY_URL=https://ch4t.buzz BUZZ_PRIVATE_KEY=$(grep '^SECRET:' /home/opc/buzz-agent-key.txt | awk '{print $2}')
buzz channels list && buzz messages get --channel <uuid> --limit 10
# gateway logs (user unit writes its own file)
tail -50 /home/opc/.hermes/logs/gateway.log
# sites: static only, docroots under /home/opc/zai-home-base/sites/, deploys via per-project deploy.sh

6. Where the previous agent (zai) made mistakes — verify, don't trust

Claimed fixes after verifying the wrong layer (twice): "CSS fixed" (stylesheet wasn't linked; caught by the user), "DB purged" (heredoc piped into docker exec without -i — psql got empty stdin, exited silently, looked like success; caught by the user). Rule: validate at the layer the user sees.
Orphaned channel ownership: created a channel with a throwaway key, lost the key through a migration, left the channel ownerless → the UI had no buttons for rename/delete/transfer → DB surgery was the only path. Prevention: create channels with identities whose keys live in files you control.
Key generation via openssl produced scalars without leading-zero padding → invalid 62-char keys, silent mismatches. Use buzz-admin generate-key.
Multiple silent no-op edits: python str.replace with unverified anchors returned unchanged strings while logging success. Assert every anchor.
The agent shell dies if its cwd is deleted (recreate the path to recover).

7. Open items (not started)

LLM key for hermes (user: hermes setup --portal)
antigravity + openfang harness connections (keys staged, guide in /home/opc/zai-home-base/buzz/onboarding.md)
dadjokes.guru + raspiinthesky.com fleet builds (design notes in repo)
helpme.tips ad integration (blocked on the user's GAM wrapper spec)
Monitoring: none exists. Presence dot is the only health signal. A dead-man's switch on the listener would be the first thing to add.

8.2 KiB Raw Blame History