65 lines
4.1 KiB
Markdown
65 lines
4.1 KiB
Markdown
|
---
|
||
|
|
name: Deploy session 2026-03-22 findings
|
||
|
|
description: Comprehensive deploy/build fixes made overnight — container issues, image tags, script improvements, remaining work
|
||
|
|
type: project
|
||
|
|
---
|
||
|
|
|
||
|
|
## Session Summary (2026-03-22 overnight)
|
||
|
|
|
||
|
|
Massive deploy infrastructure overhaul across all 5 nodes (.228, .198, Arch 1/2/3).
|
||
|
|
|
||
|
|
### Fixed in deploy-tailscale.sh
|
||
|
|
- **Image tags**: Bitcoin Knots `28.1` (not `v28.1`), BTCPay `1.13.7` (not `1.14.5`), SearXNG `2026.3.20-6c7e9c197`
|
||
|
|
- **Removed Immich** (3 containers) and **Penpot** (5 containers) from deploy + build
|
||
|
|
- **Fedimint**: `FM_REL_NOTES_ACK=0_4_xyz` env var (NOT `FM_SKIP_REL_NOTES_ACK` or `FM_REQ_RELEASE_NOTES_ACK_V0_4`)
|
||
|
|
- **Fedimint-gateway**: `--password` instead of `--bcrypt-password-hash` (v0.5.1 CLI change)
|
||
|
|
- **FileBrowser**: added `--cap-add NET_BIND_SERVICE` for port 80 binding
|
||
|
|
- **SearXNG**: added `/var/lib/archipelago/searxng:/etc/searxng` volume mount + caps
|
||
|
|
- **Postgres**: pinned to `postgres:15` (data initialized with 15, incompatible with 16)
|
||
|
|
- **Migration**: one-time flag file `/var/lib/archipelago/.rootless-migrated`
|
||
|
|
- **Recreate-if-broken pattern**: containers that exist but are stopped get deleted and recreated
|
||
|
|
- **Arch 2 hostname**: fixed from hardcoded hostname to `$TAILSCALE_ARCH2`
|
||
|
|
- **Custom UI images**: graceful skip if not available, source extracted to repo (`docker/bitcoin-ui/`, `docker/electrs-ui/`)
|
||
|
|
- **AIUI tar xattr**: silenced with `--no-xattrs` (only in deploy-tailscale.sh, NOT deploy-to-target.sh yet)
|
||
|
|
- **Nginx MIME warning**: removed `text/html` from `sub_filter_types`
|
||
|
|
|
||
|
|
### Added
|
||
|
|
- `--fleet` flag in deploy-to-target.sh: deploys .228 → .198 → Arch 1/2/3
|
||
|
|
- `--both` lock fix: releases lock before recursive `--live` call
|
||
|
|
- Container verification step (Step 26b): restarts exited containers, fixes permissions, checks Tor
|
||
|
|
- IndeedHub backend stack rebuilt on .228 (7 containers)
|
||
|
|
- IndeedHub nginx patched with direct IPs (podman DNS doesn't work with nginx resolver)
|
||
|
|
|
||
|
|
### Frontend changes
|
||
|
|
- Replaced Immich with FileBrowser on Setup homescreen (`goals.ts`, `EasyHome.vue`)
|
||
|
|
- `MEMPOOL_API_IMAGE` renamed to `MEMPOOL_BACKEND_IMAGE` in image-versions.sh
|
||
|
|
- Nextcloud downgraded from 30 to 29 (one major version upgrade at a time)
|
||
|
|
|
||
|
|
### Session 2 fixes (same day)
|
||
|
|
|
||
|
|
**Critical pattern found: Container credential mismatches**
|
||
|
|
- Deploy generates random passwords stored in `secrets/`. MariaDB/Postgres only use env vars on FIRST init — subsequent restarts ignore them. Container recreation with new passwords → auth failures → crash loops.
|
||
|
|
- 50,000+ cumulative container restarts across fleet from this single root cause.
|
||
|
|
|
||
|
|
**Fixes applied to all nodes:**
|
||
|
|
1. LND: `lnd.conf` rpcpass synced from `secrets/bitcoin-rpc-password` (was hardcoded `archipelago123`)
|
||
|
|
2. MariaDB mempool: data dirs wiped + reinitialized (password mismatch unrecoverable)
|
||
|
|
3. BTCPay Postgres: `ALTER USER` to sync password with secrets
|
||
|
|
4. FileBrowser: `--user 0:0` instead of `--cap-add NET_BIND_SERVICE` (rootless port 80 fix)
|
||
|
|
5. Nextcloud: same `--user 0:0` fix
|
||
|
|
6. Tailscale container on .228: removed (2,685 restarts — unauthenticated, host already has TS)
|
||
|
|
|
||
|
|
**Deploy script fixes:**
|
||
|
|
- `deploy-tailscale.sh`: LND config always synced before start, `eval "$DB_PASSWORDS"` → safe individual reads, MariaDB password sync step, filebrowser `--user 0:0`
|
||
|
|
- `deploy-to-target.sh`: LND stale config check now compares passwords (not just cookie/localhost), filebrowser `--user 0:0`
|
||
|
|
|
||
|
|
**Rootless port 80 rule**: Containers binding port 80 MUST use `--user 0:0`. `NET_BIND_SERVICE` cap doesn't work in rootless (UID 0 → host 100000, unprivileged).
|
||
|
|
|
||
|
|
### Remaining issues for next session
|
||
|
|
- **Vaultwarden exit 101** on Arch 2 — likely corrupted SQLite DB
|
||
|
|
- **PhotoPrism storage permission** on Arch 1 — file creation fails despite correct ownership
|
||
|
|
- **Arch 3 resource contention** — 7.3GB RAM, load 14, 28 containers. May need to reduce container count.
|
||
|
|
- **Health checks missing** on most containers (only filebrowser/jellyfin have them)
|
||
|
|
- **Tar xattr spam** in deploy-to-target.sh (fixed in deploy-tailscale.sh only)
|
||
|
|
- **IndeedHub nginx IPs are ephemeral** — need re-patch after container restart
|