The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three stacked bugs made "switch version" silently fail / crash-loop, and
the data-access mismatch corrupted a node's index during recovery attempts.
Backend renderer:
- sync_quadlet_unit ignored the per-app pinned version and re-rendered the
quadlet with the manifest's :latest every reconcile tick, reverting any
switch. Factor the install-time catalog/pin resolution into a shared
resolve_catalog_image() and call it in BOTH install_fresh and
sync_quadlet_unit.
- The renderer folded manifest `entrypoint: ["sh","-lc"]` into Exec=, which
only worked when the image entrypoint was a passthrough shell wrapper. The
versioned images use ENTRYPOINT ["bitcoind"], so Exec=sh -lc ... became
`bitcoind sh -lc ...` and crash-looped. Emit a real Entrypoint= override;
exec_changed now also compares Entrypoint=.
Images:
- Build all bitcoin images (Core + Knots, every version) as container-root
(USER removed) like the legacy :latest image. Chain data is owned by the
data_uid (container uid 102); root reads it via CAP_DAC_OVERRIDE (granted in
the manifest). A non-root USER (the previous uid 1000) can't read existing
chain data → "Error initializing block database". Still fully rootless:
container-root maps to the unprivileged host service user.
Catalog:
- bitcoin-knots versions[]: 29.3.knots20260508/20260507/20260210 +
29.2.knots20251110, "latest" tracking newest.
- bitcoin-core versions[]: add 29.2 + a "latest" entry. All images rebuilt
root and published to the mirror.
Frontend:
- AppSidebar version dropdown: rename the latest option to "Always use the
latest version" (no v prefix), fix right padding, and guarantee the current
selection matches a real option (was rendering blank).
- New InstallVersionModal: full-screen version chooser shown from the App
Store / Discover install button for multi-version apps (Bitcoin Knots/Core),
app icon + "Install <name>", latest pre-selected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Snapshots the in-flight hardening work so subsequent reconcile/Quadlet
phases land on a clean before/after diff.
Changes:
- core/container/src/podman_client.rs: image_uses_insecure_registry()
whitelist for the OVH (146.59.87.168:3000) and legacy Hetzner
(23.182.128.160:3000) HTTP mirrors; podman_network_settings() lifts
custom networks into the Networks map so containers can join them.
- core/archipelago/src/container/prod_orchestrator.rs:
ensure_container_network() creates per-manifest networks on demand;
apply_data_uid() now goes through host_sudo for mkdir -p + chown so
bind-mount roots get created and chowned without password prompts.
- core/archipelago/src/api/rpc/package/{install,update,stacks}.rs:
podman pull adds --tls-verify=false only for whitelisted registries.
- core/archipelago/src/bootstrap.rs: removes stale dev-mode systemd
override on startup (live nodes carried it from old installers).
- core/archipelago/src/config.rs: ignore ARCHIPELAGO_DEV_MODE in prod
binaries — it had been silently rerouting volumes to /tmp.
- apps/bitcoin-{core,knots}/manifest.yml: locate bitcoind at runtime
so image-layout differences don't break entrypoint.
- scripts/app-catalog-image-smoke-test.py: production catalog/image
smoke test that probes a target node before users click Install.
- .gitignore: cover .codex, .pnpm-store, __pycache__, *.bak.
Removes filebrowser.rs.bak and two stale catalog.json.bak files
(verified identical to live counterparts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
