HANDOFF.md: engineer brief — inventory, credentials map, known issues, debugging playbook, zai mistake log
This commit is contained in:
zai-agent
parent
9a2a7189a6
commit
c71f9642cd
1 changed files with 147 additions and 0 deletions
147
HANDOFF.md
Normal file
147
HANDOFF.md
Normal file
|
|
@ -0,0 +1,147 @@
|
||||||
|
# HANDOFF BRIEF — djg-oracle-sl Buzz + sites deployment
|
||||||
|
|
||||||
|
**Audience:** senior engineers taking over this stack. Written by the ZCode
|
||||||
|
agent that built it, including its own mistakes. Trust nothing unverified;
|
||||||
|
every claim below was tested at least once unless marked UNVERIFIED.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. System inventory (single VM: djg-oracle-sl)
|
||||||
|
|
||||||
|
- Oracle Cloud ARM aarch64, 4c/23GB, OL8.10, public IP **144.24.30.131**.
|
||||||
|
SELinux **enforcing**. cgroup **v1** (blocks k3s ≥1.35 — do NOT upgrade k3s
|
||||||
|
past 1.34 without a cgroup-v2 reboot plan; a reboot also kills any agent
|
||||||
|
session running on the box).
|
||||||
|
- **k3s v1.34.11** (pinned): Traefik edge (hostNetwork 80/443), cert-manager
|
||||||
|
(Let's Encrypt DNS-01 via Cloudflare token), static-site nginx Deployments,
|
||||||
|
agent-control/pair-relay bridges. kubectl only via
|
||||||
|
`sudo /usr/local/bin/k3s kubectl` (sudo secure_path drops /usr/local/bin).
|
||||||
|
- **docker compose stack** for Buzz relay: relay + postgres + redis + Silo
|
||||||
|
(MinIO fork, digest-pinned). Compose dir: /home/opc/buzz/deploy/compose
|
||||||
|
(.env = secrets, 600).
|
||||||
|
- **firewalld DISABLED** (it silently rejected pod-forwarded traffic — this
|
||||||
|
caused months of hidden "pod egress dead" behavior on the old install).
|
||||||
|
OCI security list (22/80/443) is the only perimeter.
|
||||||
|
- **coredns Corefile patched** to `forward . 1.1.1.1 8.8.8.8` because OCI's
|
||||||
|
169.254.169.254 resolver is link-local and unreachable from pods. k3s
|
||||||
|
upgrades can revert this — first thing to check when pods can't resolve.
|
||||||
|
- Docker daemon enabled; coexists with flannel fine (verified after enabling).
|
||||||
|
|
||||||
|
## 2. Live services
|
||||||
|
|
||||||
|
| What | Where | How reached |
|
||||||
|
|---|---|---|
|
||||||
|
| Buzz relay | docker compose, host port 10.0.0.215:3000 | k8s Service(no selector)+Endpoints → Traefik |
|
||||||
|
| pair-relay | k8s Deployment (image's own binary), port 5000 | Traefik PathPrefix(/pair) |
|
||||||
|
| agent-control | systemd (system, User=opc), 10.0.0.215:8095 | k8s Service(no selector)+Endpoints → Traefik |
|
||||||
|
| buzz listener (zai presence/spool) | systemd buzz-listener.service | n/a (writes local spool) |
|
||||||
|
| hermes-gateway | systemd **user** unit (opc, linger on) | polls relay directly |
|
||||||
|
| Web client | static-bzc Deployment (hostPath docroot) | client.ch4t.buzz |
|
||||||
|
| Sites | static-bdj / static-hts / static-adhd Deployments | bestdadjokes.lol / helpme.tips / myadhd.dev |
|
||||||
|
|
||||||
|
Domains: ch4t.buzz (relay, /pair, agent control), client.ch4t.buzz (web),
|
||||||
|
bestdadjokes.lol, helpme.tips, myadhd.dev. Wildcard A * exists for ch4t.buzz.
|
||||||
|
DNS via Cloudflare API; token at ~/.cloudflare-token (600).
|
||||||
|
|
||||||
|
## 3. Secrets & keys map (all 600, none committed)
|
||||||
|
|
||||||
|
- /home/opc/.cloudflare-token — CF API token (zones: bestdadjokes.lol,
|
||||||
|
myadhd.dev, helpme.tips, ch4t.buzz)
|
||||||
|
- /home/opc/buzz-agent-key.txt — zai identity (relay admin)
|
||||||
|
- /home/opc/buzz-hermes-key.txt, buzz-antigravity-key.txt,
|
||||||
|
buzz-openfang-key.txt — staged harness identities (relay members)
|
||||||
|
- /home/opc/buzz-owner-key.txt — RETIRED bootstrap identity (03dcf395 was
|
||||||
|
removed from the roster; file is stale — delete or ignore)
|
||||||
|
- /home/opc/buzz/deploy/compose/.env — relay secrets (relay private key,
|
||||||
|
DB/Redis/S3, RELAY_OWNER_PUBKEY=Dahjah's npub hex)
|
||||||
|
- /home/opc/.hermes/.env — hermes secrets + GATEWAY_ALLOW_ALL_USERS=true
|
||||||
|
- /home/opc/.buzz-pending.json — reactions awaiting cleanup (listener state)
|
||||||
|
|
||||||
|
## 4. Known issues, ranked (what to look at first)
|
||||||
|
|
||||||
|
### A. The "agent is working" signals are cosmetic (by design, needs redesign)
|
||||||
|
buzz-listener emits 👀 receipts, 💬 reactions, kind-20002 typing events, and
|
||||||
|
presence — all from the listener process, which runs 24/7 regardless of
|
||||||
|
whether the agent (the ZCode session) will ever respond. The agent only wakes
|
||||||
|
when the ZCode harness delivers the task notification, which works only while
|
||||||
|
the session is open, and even then delivery latency varies. The user correctly
|
||||||
|
identified this: green dot + typing ≠ agent working. Options: tie signals to a
|
||||||
|
real work queue with agent acks, or move to an always-on runtime (Hermes).
|
||||||
|
|
||||||
|
### B. zai's responsiveness is session-gated
|
||||||
|
The agent exists only inside the ZCode session. Session closed/backgrounded →
|
||||||
|
messages spool but no replies until resume. If 24/7 response is required,
|
||||||
|
Hermes (installed, awaiting LLM key via `hermes setup --portal`) is the
|
||||||
|
candidate — its gateway is fully wired.
|
||||||
|
|
||||||
|
### C. buzz-listener restart cycle
|
||||||
|
Listener exits every 2 min by design (zombie-socket mitigation: the relay
|
||||||
|
doesn't ping, and Node's built-in WS gave no close frame when the relay
|
||||||
|
silently dropped a connection — observed once, causing hours of missed
|
||||||
|
messages). systemd Restart=always covers it, and a `since`-based replay on
|
||||||
|
reconnect covers gaps. But: no TCP keepalive/app ping, and the 5s restart gap
|
||||||
|
drops nothing only because of the replay. A proper fix = real WS ping/pong
|
||||||
|
liveness. Also: reactions dedupe via /home/opc/.buzz-reacted (file, append-only).
|
||||||
|
|
||||||
|
### D. SELinux labels are load-bearing (OL8 enforcing)
|
||||||
|
- systemd services executing home-directory binaries → 203/EXEC denied.
|
||||||
|
Fixed by semanage fcontext: hermes venv + uv python → bin_t (persistent).
|
||||||
|
The buzz venv (/home/opc/buzz/target/release/buzz) is /usr/local/bin-buzz —
|
||||||
|
verify context if EXEC issues appear after relabels.
|
||||||
|
- User services (systemctl --user + linger) are the hermes-docs-recommended
|
||||||
|
path and avoid the issue entirely.
|
||||||
|
|
||||||
|
### E. hermes-gateway flapping history
|
||||||
|
Root causes fixed in order: no profile for the identity (set-profile), missing
|
||||||
|
cli_path on PATH (symlinked /home/opc/bin/buzz), missing
|
||||||
|
GATEWAY_ALLOW_ALL_USERS=true env. If it flaps again: ~/.hermes/logs/gateway.log
|
||||||
|
has the real logs (journald often shows nothing for the user unit).
|
||||||
|
|
||||||
|
### F. Upstream drift
|
||||||
|
- block/buzz moves fast (relay image :main, unpinned). Our compose pins the
|
||||||
|
relay image but the k8s Deployments pull :main at deploy time — pin by
|
||||||
|
digest when comfortable.
|
||||||
|
- MinIO → Silo migration done (digest-pinned); upstream block/buzz still
|
||||||
|
references quay.io MinIO — if they migrate, mirror their choice.
|
||||||
|
|
||||||
|
## 5. Debugging playbook (commands that actually work)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# relay health + NIP-11
|
||||||
|
curl -s -H 'Accept: application/nostr+json' https://ch4t.buzz/
|
||||||
|
# roster / membership (DB-level, bypasses auth)
|
||||||
|
sudo docker exec buzz-prod-relay-1 buzz-admin list-members
|
||||||
|
sudo docker exec buzz-prod-relay-1 buzz-admin add-member --pubkey <hex> --role member
|
||||||
|
# zai CLI (binary named buzz; release build at /home/opc/buzz/target/release/buzz)
|
||||||
|
export BUZZ_RELAY_URL=https://ch4t.buzz BUZZ_PRIVATE_KEY=$(grep '^SECRET:' /home/opc/buzz-agent-key.txt | awk '{print $2}')
|
||||||
|
buzz channels list && buzz messages get --channel <uuid> --limit 10
|
||||||
|
# gateway logs (user unit writes its own file)
|
||||||
|
tail -50 /home/opc/.hermes/logs/gateway.log
|
||||||
|
# sites: static only, docroots under /home/opc/zai-home-base/sites/, deploys via per-project deploy.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Where the previous agent (zai) made mistakes — verify, don't trust
|
||||||
|
|
||||||
|
1. Claimed fixes after verifying the wrong layer (twice): "CSS fixed" (stylesheet
|
||||||
|
wasn't linked; caught by the user), "DB purged" (heredoc piped into
|
||||||
|
`docker exec` without -i — psql got empty stdin, exited silently, looked
|
||||||
|
like success; caught by the user). Rule: validate at the layer the user sees.
|
||||||
|
2. Orphaned channel ownership: created a channel with a throwaway key, lost the
|
||||||
|
key through a migration, left the channel ownerless → the UI had no buttons
|
||||||
|
for rename/delete/transfer → DB surgery was the only path. Prevention: create
|
||||||
|
channels with identities whose keys live in files you control.
|
||||||
|
3. Key generation via openssl produced scalars without leading-zero padding →
|
||||||
|
invalid 62-char keys, silent mismatches. Use `buzz-admin generate-key`.
|
||||||
|
4. Multiple silent no-op edits: python str.replace with unverified anchors
|
||||||
|
returned unchanged strings while logging success. Assert every anchor.
|
||||||
|
5. The agent shell dies if its cwd is deleted (recreate the path to recover).
|
||||||
|
|
||||||
|
## 7. Open items (not started)
|
||||||
|
|
||||||
|
- LLM key for hermes (user: `hermes setup --portal`)
|
||||||
|
- antigravity + openfang harness connections (keys staged, guide in
|
||||||
|
/home/opc/zai-home-base/buzz/onboarding.md)
|
||||||
|
- dadjokes.guru + raspiinthesky.com fleet builds (design notes in repo)
|
||||||
|
- helpme.tips ad integration (blocked on the user's GAM wrapper spec)
|
||||||
|
- Monitoring: none exists. Presence dot is the only health signal. A dead-man's
|
||||||
|
switch on the listener would be the first thing to add.
|
||||||
Loading…
Reference in a new issue