From 4d1df4a31984bddbbb029abfa6454f4fff24cf88 Mon Sep 17 00:00:00 2001 From: Dorian Date: Tue, 24 Mar 2026 18:06:57 +0000 Subject: [PATCH] docs: update deploy session memory with session 3 fixes Co-Authored-By: Claude Opus 4.6 (1M context) --- .../project_deploy_session_2026_03_22.md | 48 ++++++++++++++++--- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/.claude/memory/project_deploy_session_2026_03_22.md b/.claude/memory/project_deploy_session_2026_03_22.md index c5d48d21..b8f0e53d 100644 --- a/.claude/memory/project_deploy_session_2026_03_22.md +++ b/.claude/memory/project_deploy_session_2026_03_22.md @@ -55,10 +55,44 @@ Massive deploy infrastructure overhaul across all 5 nodes (.228, .198, Arch 1/2/ **Rootless port 80 rule**: Containers binding port 80 MUST use `--user 0:0`. `NET_BIND_SERVICE` cap doesn't work in rootless (UID 0 → host 100000, unprivileged). -### Remaining issues for next session -- **Vaultwarden exit 101** on Arch 2 — likely corrupted SQLite DB -- **PhotoPrism storage permission** on Arch 1 — file creation fails despite correct ownership -- **Arch 3 resource contention** — 7.3GB RAM, load 14, 28 containers. May need to reduce container count. -- **Health checks missing** on most containers (only filebrowser/jellyfin have them) -- **Tar xattr spam** in deploy-to-target.sh (fixed in deploy-tailscale.sh only) -- **IndeedHub nginx IPs are ephemeral** — need re-patch after container restart +### Session 3 fixes (2026-03-22 to 2026-03-24) + +**Additional container fixes applied live:** +- PhotoPrism: recreated with proper `/photoprism/storage`, `/photoprism/originals`, `/photoprism/import` volume mounts (all 3 nodes) +- Vaultwarden/Jellyfin: recreated with `--user 0:0` + health checks (Arch 1/2) +- Nextcloud: downgraded image to v29 (data initialized with v28, can't skip to v30) +- Fedimint: upgraded v0.5.1 → v0.10.0 on all Tailscale nodes +- Fedimint-gateway: bcrypt hash passed via file mount (shell escaping workaround) +- SearXNG: recreated with proper caps on Arch 2 +- Arch 3 right-sized: stopped immich (3), jellyfin, vaultwarden, nbxplorer (7.3GB RAM) + +**Deploy script improvements (6 commits pushed):** +1. `d37165ca` — Credential sync, health checks, rootless port binding +2. `f5714a5b` — Fleet deploy falls back to Tailscale when LAN unreachable, `--all` alias +3. `028248df` — Suppress tar xattr spam in AIUI deploy (`--no-xattrs`) +4. `f5802f9e` — Fix LND config SSH escaping, Tailscale fallback for BUILD_SOURCE +5. `06d85e1d` — Fix health check escaping for SSH heredoc (`--health-cmd 'cmd'` not `"cmd"`) +6. `a7920de8` — Correct health check endpoints (fedimint→8175, nextcloud→`/`, filebrowser→`/`) + +**Health checks added to deploy-tailscale.sh:** +- 25 containers now have `--health-cmd` in deploy-tailscale.sh (was zero) +- Key corrections: fedimint checks port 8175 (UI) not 8174 (websocket), nextcloud/filebrowser check `/` not custom endpoints + +**Fleet status at end of session:** + +| Node | Status | Notes | +|------|--------|-------| +| .228 | 36/36, 0 unhealthy, load 1.0 | Fully stable | +| Arch 1 | 25/25, 0 unhealthy, load 0.5 | Fully stable | +| Arch 2 | 25/25, 0 unhealthy, load 0.2 | Fully stable | +| Arch 3 | 24/28, 0 unhealthy, load 7.7 | Right-sized for 7.3GB RAM, Bitcoin IBD at 97.8% | +| .198 | Bitcoin chain data empty (4KB) | Needs full IBD — will take days. Not pruned. | + +### Remaining for next session +- **.198**: Bitcoin doing full IBD from scratch (chain data was lost/empty). No prune flag set. Will take days. +- **Arch 3**: Bitcoin IBD was at 97.8% — check if complete, then start LND/nbxplorer +- **Tor config Python syntax errors** in deploy-to-target.sh step 33 (cosmetic, falls back to system Tor) +- **deploy-to-target.sh** still missing health checks (only deploy-tailscale.sh has them) +- **first-boot-containers.sh** needs same rootless fixes (filebrowser `--user 0:0`, credential sync) +- **Fedimint guardian setup** not done on any node — all in "Setup UI" mode +- User needs to `git pull && ./scripts/deploy-to-target.sh --all` to deploy latest fixes to Tailscale nodes