Production edge: Traefik v2.11 (entrypoint TLS), cert-manager LE certs live on both domains; in-place docroot deploys

README.md

@@ -1,38 +1,59 @@
 # zai-home-base — revenue projects on djg-oracle-sl
 
-Workspace for projects built to generate revenue. All sites are served by a
-shared nginx edge DaemonSet in k3s (hostPort 80) reading from `sites/`.
+Workspace for revenue-generating projects. Cluster edge: **Traefik v2.11**
+(hostNetwork 80/443) + **cert-manager** (Let's Encrypt DNS-01 via Cloudflare).
+Sites are static nginx Deployments fed by hostPath docroots in `sites/`.
+
+## Live sites
+
+- **bestdadjokes.lol** — dad-joke content site (Track 1; ads play via user's AdX/MCM)
+- **myadhd.dev** — placeholder; ADHD micro-tools product coming (Track 2, Paddle/Stripe later)
 
 ## Layout
 
-- `bdj/` — bestdadjokes.lol (dad jokes content site, Track 1)
-  - `content/jokes.json` — the dataset (append-only; joke ids = array index)
-  - `build.py` — stdlib-only static site generator (Python 3.6)
-  - `static/` — css/js/favicon (jokes-data.js is generated, don't hand-edit)
-  - `site/` — build output (gitignored)
-  - `nginx/conf.d/` — per-site nginx vhosts (mounted into edge pod)
-  - `k8s/nginx-edge.yaml` — shared edge DaemonSet (hostNetwork, port 80)
-  - `deploy.sh` — build + atomic swap + reload + verify
-- `sites/` — live docroots served by the edge pod (generated, gitignored)
-- `nginx-ingress.yaml` (in /home/opc) — leftover demo manifest, unused
+- `bdj/` — bestdadjokes.lol project
+  - `content/jokes.json` — dataset, APPEND-ONLY (joke id = array index)
+  - `build.py` — stdlib-only static generator (Python 3.6! no walrus, no dict-merge `|`)
+  - `static/` — css/js/favicon (`jokes-data.js` is generated, don't hand-edit)
+  - `k8s/traefik.yaml` — edge DaemonSet (hostNetwork, RBAC incl. endpointslices+nodes)
+  - `k8s/letsencrypt-issuer.yaml` — ClusterIssuer, DNS-01 via CF token Secret
+  - `k8s/static-sites.yaml` — per-site Deployment+Service+Ingress+Certificate
+  - `deploy.sh` — build + in-place rsync to docroot (no dir swap — see gotchas)
+- `adhd/site/index.html` — myadhd.dev placeholder (copied to `sites/myadhd.dev/`)
+- `sites/` — live docroots (generated, gitignored)
 
-## Deploy a content change
+## Deploy content
 
     cd bdj && ./deploy.sh
 
-## Server facts
+New site = docroot + copy the Deployment/Service/Ingress/Certificate pattern in
+`bdj/k8s/static-sites.yaml` + a DNS record (Cloudflare API, token in
+`~/.cloudflare-token`, zones bestdadjokes.lol=0cd16b5205725d310366859b3352a189,
+myadhd.dev=00fa64c074d7162b7616a4a3dc42ba77).
 
-- VM: djg-oracle-sl (Oracle Cloud ARM, 4c/23GB), public IP 144.24.30.131
-- k3s single node (control-plane taint — pods need the toleration in k8s/nginx-edge.yaml)
-- kubectl: `sudo /usr/local/bin/k3s kubectl` (no standalone kubeconfig for opc)
-- Local test: `curl -H 'Host: bestdadjokes.lol' http://127.0.0.1/`
-- Inbound 80/443 must be open in the OCI VCN security list (cloud console side)
-- DNS: user manages Cloudflare; site goes live with proxied A record → 144.24.30.131
+## Server facts & gotchas (hard-won)
 
-## Roadmap (bdj)
+- VM: djg-oracle-sl, Oracle Linux 8.10, **ARM** 4c/23GB, public IP 144.24.30.131
+- **k3s pinned v1.34.11**: newest kubelet (1.36) refuses cgroup v1; OL8 boots cgroup v1
+  and a reboot to switch would kill the agent session. Revisit only with user OK.
+- kubectl: `sudo /usr/local/bin/k3s kubectl` (sudo secure_path drops /usr/local/bin)
+- **firewalld is DISABLED** (it silently rejected pod-forwarded traffic → all pod
+  egress dead, "No route to host"). OCI security list (22/80/443) is the perimeter.
+- **coredns Corefile patched**: `forward . 1.1.1.1 8.8.8.8` because OCI's
+  169.254.169.254 resolver is link-local and unreachable from pods. A k3s upgrade
+  may revert this — re-patch if pods can't resolve DNS.
+- **Traefik gotcha**: without `--entrypoints.websecure.http.tls=true`, TLS routers
+  register but never match → every HTTPS request 404s while plaintext works.
+- **hostPath gotcha**: never atomic-swap docroot dirs (mv breaks the bind by inode);
+  rsync in place. If a pod serves 403/empty after content ops, rollout-restart it.
+- Local test: `curl -sk --resolve bestdadjokes.lol:443:127.0.0.1 https://bestdadjokes.lol/`
+- Shell rule: never `cd` into a directory a command might delete (agent shell cwd
+  breaks and needs manual recovery).
 
-1. DNS live → submit to Google Search Console (HTML meta verification)
-2. Social content (needs user-provided accounts) — daily share-card images
-3. Ad integration: user's MCM/AdX stack; ad slots are marked `<!-- AD_SLOT_* -->`
-4. Expand dataset daily; long-tail topic pages ("dad jokes about coffee")
-5. Phase 2 (myadhd.dev): ADHD micro-tools + Paddle/Stripe when user ready
+## Roadmap
+
+1. Google Search Console (user creates property; HTML-meta verification token → I add)
+2. Social content pipeline (needs user accounts: X/Pinterest/Bluesky/Reddit)
+3. Ads: site into user's MCM/AdX; ad slots marked `<!-- AD_SLOT_* -->` in build.py
+4. Daily content growth + long-tail topic pages; then paid-traffic arb experiments
+5. myadhd.dev build-out (client-side tools, Paddle/Stripe when user ready)

bdj/deploy.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
-# Build sites and sync docroots. Routing/TLS are handled by Traefik + cert-manager
-# in the cluster, so content deploys need no pod reloads or restarts.
+# Build sites and sync docroots IN PLACE (no dir swap: hostPath mounts bind to
+# the directory inode, so replacing the dir would leave pods serving a stale
+# path). Routing/TLS are handled by Traefik + cert-manager; no reloads needed.
 set -euo pipefail
 cd "$(dirname "$0")"
 BASE=/home/opc/zai-home-base
@@ -8,16 +9,11 @@ BASE=/home/opc/zai-home-base
 echo "==> building site"
 python3 build.py
 
-mkdir -p "$BASE/sites"
-rm -rf "$BASE/sites/bestdadjokes.new"
-cp -r site "$BASE/sites/bestdadjokes.new"
+mkdir -p "$BASE/sites/bestdadjokes"
 if command -v getenforce >/dev/null 2>&1 && [ "$(getenforce)" = "Enforcing" ]; then
-  chcon -Rt container_file_t "$BASE/sites/bestdadjokes.new" 2>/dev/null || true
+  chcon -Rt container_file_t "$BASE/sites/bestdadjokes" 2>/dev/null || true
 fi
-rm -rf "$BASE/sites/bestdadjokes.old"
-[ -d "$BASE/sites/bestdadjokes" ] && mv "$BASE/sites/bestdadjokes" "$BASE/sites/bestdadjokes.old"
-mv "$BASE/sites/bestdadjokes.new" "$BASE/sites/bestdadjokes"
-rm -rf "$BASE/sites/bestdadjokes.old"
+rsync -a --delete site/ "$BASE/sites/bestdadjokes/"
 
 echo "==> verifying through Traefik (origin)"
 sleep 1

bdj/k8s/traefik.yaml

@@ -1,5 +1,8 @@
-# Traefik v3 as the cluster edge: hostNetwork 80/443, global http->https redirect.
-# TLS certificates are provisioned by cert-manager into Ingress secrets.
+# Traefik v2.11 LTS as the cluster edge: hostNetwork 80/443, global http->https redirect.
+# TLS: entrypoint terminates TLS; certificates provisioned by cert-manager
+# (Let's Encrypt DNS-01 via Cloudflare) into the Ingress TLS secrets.
+# NOTE: --entrypoints.websecure.http.tls=true is REQUIRED — without it TLS
+# routers never engage the mux and every HTTPS request 404s.
 apiVersion: v1
 kind: ServiceAccount
 metadata:
@@ -12,7 +15,10 @@ metadata:
   name: traefik
 rules:
   - apiGroups: [""]
-    resources: ["services", "endpoints", "secrets", "namespaces"]
+    resources: ["services", "endpoints", "secrets", "namespaces", "nodes"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["discovery.k8s.io"]
+    resources: ["endpointslices"]
     verbs: ["get", "list", "watch"]
   - apiGroups: ["networking.k8s.io"]
     resources: ["ingresses", "ingressclasses"]
@@ -64,16 +70,16 @@ spec:
           effect: NoSchedule
       containers:
         - name: traefik
-          image: traefik:v3.3
+          image: traefik:v2.11.2
           args:
             - --providers.kubernetesingress
+            - --providers.kubernetescrd=false
             - --entrypoints.web.address=:80
             - --entrypoints.web.http.redirections.entrypoint.to=websecure
             - --entrypoints.web.http.redirections.entrypoint.scheme=https
             - --entrypoints.websecure.address=:443
-            - --ping=true
-            - --ping.entrypoint=health
-            - --entrypoints.health.address=:8082
+            - --entrypoints.websecure.http.tls=true
+            - --log.level=WARN
           ports:
             - containerPort: 80
               hostPort: 80
@@ -89,13 +95,19 @@ spec:
               memory: 256Mi
           readinessProbe:
             httpGet:
-              path: /ping
-              port: 8082
-            initialDelaySeconds: 3
-            periodSeconds: 10
+              path: /
+              port: 80
+              httpHeaders:
+                - name: Host
+                  value: bestdadjokes.lol
+            initialDelaySeconds: 5
+            periodSeconds: 15
           livenessProbe:
             httpGet:
-              path: /ping
-              port: 8082
-            initialDelaySeconds: 10
+              path: /
+              port: 80
+              httpHeaders:
+                - name: Host
+                  value: bestdadjokes.lol
+            initialDelaySeconds: 15
             periodSeconds: 30

bdj/nginx/README.md

@@ -1,2 +0,0 @@
-# Placeholder: the hand-rolled nginx edge was removed in favor of Traefik + cert-manager.
-# Vhosts now live as Ingress resources in bdj/k8s/static-sites.yaml.

bdj/nginx/certs/README.md

@@ -1,2 +0,0 @@
-# Placeholder: self-signed origin certs were removed in favor of cert-manager
-# issuing Let's Encrypt certificates into k8s Secrets (see bdj/k8s/letsencrypt-issuer.yaml).