HANDOFF.md: engineer brief — inventory, credentials map, known issues, debugging playbook, zai mistake log
This commit is contained in:
zai-agent
parent
9a2a7189a6
commit
c71f9642cd
1 changed files with 147 additions and 0 deletions
147
HANDOFF.md
Normal file
147
HANDOFF.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# HANDOFF BRIEF — djg-oracle-sl Buzz + sites deployment
|
||||
|
||||
**Audience:** senior engineers taking over this stack. Written by the ZCode
|
||||
agent that built it, including its own mistakes. Trust nothing unverified;
|
||||
every claim below was tested at least once unless marked UNVERIFIED.
|
||||
|
||||
---
|
||||
|
||||
## 1. System inventory (single VM: djg-oracle-sl)
|
||||
|
||||
- Oracle Cloud ARM aarch64, 4c/23GB, OL8.10, public IP **144.24.30.131**.
|
||||
SELinux **enforcing**. cgroup **v1** (blocks k3s ≥1.35 — do NOT upgrade k3s
|
||||
past 1.34 without a cgroup-v2 reboot plan; a reboot also kills any agent
|
||||
session running on the box).
|
||||
- **k3s v1.34.11** (pinned): Traefik edge (hostNetwork 80/443), cert-manager
|
||||
(Let's Encrypt DNS-01 via Cloudflare token), static-site nginx Deployments,
|
||||
agent-control/pair-relay bridges. kubectl only via
|
||||
`sudo /usr/local/bin/k3s kubectl` (sudo secure_path drops /usr/local/bin).
|
||||
- **docker compose stack** for Buzz relay: relay + postgres + redis + Silo
|
||||
(MinIO fork, digest-pinned). Compose dir: /home/opc/buzz/deploy/compose
|
||||
(.env = secrets, 600).
|
||||
- **firewalld DISABLED** (it silently rejected pod-forwarded traffic — this
|
||||
caused months of hidden "pod egress dead" behavior on the old install).
|
||||
OCI security list (22/80/443) is the only perimeter.
|
||||
- **coredns Corefile patched** to `forward . 1.1.1.1 8.8.8.8` because OCI's
|
||||
169.254.169.254 resolver is link-local and unreachable from pods. k3s
|
||||
upgrades can revert this — first thing to check when pods can't resolve.
|
||||
- Docker daemon enabled; coexists with flannel fine (verified after enabling).
|
||||
|
||||
## 2. Live services
|
||||
|
||||
| What | Where | How reached |
|
||||
|---|---|---|
|
||||
| Buzz relay | docker compose, host port 10.0.0.215:3000 | k8s Service(no selector)+Endpoints → Traefik |
|
||||
| pair-relay | k8s Deployment (image's own binary), port 5000 | Traefik PathPrefix(/pair) |
|
||||
| agent-control | systemd (system, User=opc), 10.0.0.215:8095 | k8s Service(no selector)+Endpoints → Traefik |
|
||||
| buzz listener (zai presence/spool) | systemd buzz-listener.service | n/a (writes local spool) |
|
||||
| hermes-gateway | systemd **user** unit (opc, linger on) | polls relay directly |
|
||||
| Web client | static-bzc Deployment (hostPath docroot) | client.ch4t.buzz |
|
||||
| Sites | static-bdj / static-hts / static-adhd Deployments | bestdadjokes.lol / helpme.tips / myadhd.dev |
|
||||
|
||||
Domains: ch4t.buzz (relay, /pair, agent control), client.ch4t.buzz (web),
|
||||
bestdadjokes.lol, helpme.tips, myadhd.dev. Wildcard A * exists for ch4t.buzz.
|
||||
DNS via Cloudflare API; token at ~/.cloudflare-token (600).
|
||||
|
||||
## 3. Secrets & keys map (all 600, none committed)
|
||||
|
||||
- /home/opc/.cloudflare-token — CF API token (zones: bestdadjokes.lol,
|
||||
myadhd.dev, helpme.tips, ch4t.buzz)
|
||||
- /home/opc/buzz-agent-key.txt — zai identity (relay admin)
|
||||
- /home/opc/buzz-hermes-key.txt, buzz-antigravity-key.txt,
|
||||
buzz-openfang-key.txt — staged harness identities (relay members)
|
||||
- /home/opc/buzz-owner-key.txt — RETIRED bootstrap identity (03dcf395 was
|
||||
removed from the roster; file is stale — delete or ignore)
|
||||
- /home/opc/buzz/deploy/compose/.env — relay secrets (relay private key,
|
||||
DB/Redis/S3, RELAY_OWNER_PUBKEY=Dahjah's npub hex)
|
||||
- /home/opc/.hermes/.env — hermes secrets + GATEWAY_ALLOW_ALL_USERS=true
|
||||
- /home/opc/.buzz-pending.json — reactions awaiting cleanup (listener state)
|
||||
|
||||
## 4. Known issues, ranked (what to look at first)
|
||||
|
||||
### A. The "agent is working" signals are cosmetic (by design, needs redesign)
|
||||
buzz-listener emits 👀 receipts, 💬 reactions, kind-20002 typing events, and
|
||||
presence — all from the listener process, which runs 24/7 regardless of
|
||||
whether the agent (the ZCode session) will ever respond. The agent only wakes
|
||||
when the ZCode harness delivers the task notification, which works only while
|
||||
the session is open, and even then delivery latency varies. The user correctly
|
||||
identified this: green dot + typing ≠ agent working. Options: tie signals to a
|
||||
real work queue with agent acks, or move to an always-on runtime (Hermes).
|
||||
|
||||
### B. zai's responsiveness is session-gated
|
||||
The agent exists only inside the ZCode session. Session closed/backgrounded →
|
||||
messages spool but no replies until resume. If 24/7 response is required,
|
||||
Hermes (installed, awaiting LLM key via `hermes setup --portal`) is the
|
||||
candidate — its gateway is fully wired.
|
||||
|
||||
### C. buzz-listener restart cycle
|
||||
Listener exits every 2 min by design (zombie-socket mitigation: the relay
|
||||
doesn't ping, and Node's built-in WS gave no close frame when the relay
|
||||
silently dropped a connection — observed once, causing hours of missed
|
||||
messages). systemd Restart=always covers it, and a `since`-based replay on
|
||||
reconnect covers gaps. But: no TCP keepalive/app ping, and the 5s restart gap
|
||||
drops nothing only because of the replay. A proper fix = real WS ping/pong
|
||||
liveness. Also: reactions dedupe via /home/opc/.buzz-reacted (file, append-only).
|
||||
|
||||
### D. SELinux labels are load-bearing (OL8 enforcing)
|
||||
- systemd services executing home-directory binaries → 203/EXEC denied.
|
||||
Fixed by semanage fcontext: hermes venv + uv python → bin_t (persistent).
|
||||
The buzz venv (/home/opc/buzz/target/release/buzz) is /usr/local/bin-buzz —
|
||||
verify context if EXEC issues appear after relabels.
|
||||
- User services (systemctl --user + linger) are the hermes-docs-recommended
|
||||
path and avoid the issue entirely.
|
||||
|
||||
### E. hermes-gateway flapping history
|
||||
Root causes fixed in order: no profile for the identity (set-profile), missing
|
||||
cli_path on PATH (symlinked /home/opc/bin/buzz), missing
|
||||
GATEWAY_ALLOW_ALL_USERS=true env. If it flaps again: ~/.hermes/logs/gateway.log
|
||||
has the real logs (journald often shows nothing for the user unit).
|
||||
|
||||
### F. Upstream drift
|
||||
- block/buzz moves fast (relay image :main, unpinned). Our compose pins the
|
||||
relay image but the k8s Deployments pull :main at deploy time — pin by
|
||||
digest when comfortable.
|
||||
- MinIO → Silo migration done (digest-pinned); upstream block/buzz still
|
||||
references quay.io MinIO — if they migrate, mirror their choice.
|
||||
|
||||
## 5. Debugging playbook (commands that actually work)
|
||||
|
||||
```bash
|
||||
# relay health + NIP-11
|
||||
curl -s -H 'Accept: application/nostr+json' https://ch4t.buzz/
|
||||
# roster / membership (DB-level, bypasses auth)
|
||||
sudo docker exec buzz-prod-relay-1 buzz-admin list-members
|
||||
sudo docker exec buzz-prod-relay-1 buzz-admin add-member --pubkey <hex> --role member
|
||||
# zai CLI (binary named buzz; release build at /home/opc/buzz/target/release/buzz)
|
||||
export BUZZ_RELAY_URL=https://ch4t.buzz BUZZ_PRIVATE_KEY=$(grep '^SECRET:' /home/opc/buzz-agent-key.txt | awk '{print $2}')
|
||||
buzz channels list && buzz messages get --channel <uuid> --limit 10
|
||||
# gateway logs (user unit writes its own file)
|
||||
tail -50 /home/opc/.hermes/logs/gateway.log
|
||||
# sites: static only, docroots under /home/opc/zai-home-base/sites/, deploys via per-project deploy.sh
|
||||
```
|
||||
|
||||
## 6. Where the previous agent (zai) made mistakes — verify, don't trust
|
||||
|
||||
1. Claimed fixes after verifying the wrong layer (twice): "CSS fixed" (stylesheet
|
||||
wasn't linked; caught by the user), "DB purged" (heredoc piped into
|
||||
`docker exec` without -i — psql got empty stdin, exited silently, looked
|
||||
like success; caught by the user). Rule: validate at the layer the user sees.
|
||||
2. Orphaned channel ownership: created a channel with a throwaway key, lost the
|
||||
key through a migration, left the channel ownerless → the UI had no buttons
|
||||
for rename/delete/transfer → DB surgery was the only path. Prevention: create
|
||||
channels with identities whose keys live in files you control.
|
||||
3. Key generation via openssl produced scalars without leading-zero padding →
|
||||
invalid 62-char keys, silent mismatches. Use `buzz-admin generate-key`.
|
||||
4. Multiple silent no-op edits: python str.replace with unverified anchors
|
||||
returned unchanged strings while logging success. Assert every anchor.
|
||||
5. The agent shell dies if its cwd is deleted (recreate the path to recover).
|
||||
|
||||
## 7. Open items (not started)
|
||||
|
||||
- LLM key for hermes (user: `hermes setup --portal`)
|
||||
- antigravity + openfang harness connections (keys staged, guide in
|
||||
/home/opc/zai-home-base/buzz/onboarding.md)
|
||||
- dadjokes.guru + raspiinthesky.com fleet builds (design notes in repo)
|
||||
- helpme.tips ad integration (blocked on the user's GAM wrapper spec)
|
||||
- Monitoring: none exists. Presence dot is the only health signal. A dead-man's
|
||||
switch on the listener would be the first thing to add.
|
||||
Loading…
Reference in a new issue