Hotfix: archipelago.service ExecStartPre now mkdirs /run/containers and
/var/lib/containers before the unit's mount-namespace setup tries to bind
them. Without this, fresh nodes that don't have /run/containers (e.g.
nodes provisioned without a prior podman session) fail at the namespace
step with:
Failed to set up mount namespacing: /run/containers: No such file or directory
Failed at step NAMESPACE spawning /bin/bash: No such file or directory
Existing nodes don't pick up systemd unit changes via OTA — they need a
one-time `systemctl edit archipelago` adding the same mkdir. ISO installs
from this version forward have the fix baked in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sync-perf tuning for bitcoin/bitcoin-core/bitcoin-knots/electrumx.
- Drop the --cpus=2 cap on bitcoin/electrumx variants. Script verification
is parallelizable; the cap halved IBD speed on 4-8 core machines.
- Bump bitcoin --memory 4g→8g so dbcache=4096 has headroom for mempool +
connection buffers + I/O. 4g was OOM-prone during heavy IBD.
- Bump electrumx --memory 1g→2g + add CACHE_MB=2048 + MAX_SEND=10MB.
- bitcoin-core CLI args gain -dbcache=4096 -par=0 -maxconnections=125.
- bitcoin-knots manifest matched (1024MB pruned / 4096MB full + par=0).
Future v2: host-RAM-aware dbcache scaling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resilience-validated release. Three full sweeps of the new resilience
harness against .228 confirm no shipstoppers.
Big user-visible:
- Bitcoin RPC auth durably correct via host-rendered nginx.conf bind-mount,
replaces fragile post-start exec that failed under restricted-cap rootless
podman ("crun: write cgroup.procs: Permission denied")
- Multi-container stack installs (indeedhub, immich, btcpay, mempool) now
emit phase events at every boundary so the progress bar advances
- Apps no longer vanish from the dashboard mid-install (absent-scanner skips
packages in transitional states)
- Indeedhub fresh installs work end-to-end (was 8500+ restart loop): five
missing env vars (DATABASE_PORT, QUEUE_HOST, QUEUE_PORT,
S3_PRIVATE_BUCKET_NAME, AES_MASTER_SECRET) added to install code
- Tailscale install fixed: --entrypoint string was being passed as a single
shell-line arg; switched to custom_args array
- Catalog cleaned of broken entries (dwn, endurain, ollama removed; nextcloud
restored on docker.io)
- Bitcoin Core update path uses correct image (was looking for nonexistent
lfg2025/bitcoin:28.4)
- ISO installs now allocate swap on the encrypted data partition
Infra:
- New resilience harness (scripts/resilience/) — black-box state-machine
tester, every app × every transition. Run before each release.
Sweep #3 final: PASS 107 / FAIL 12 / SKIP 14. The 12 fails are 1 cosmetic
(homeassistant trusted_hosts), 8 harness/timing false-positives, and 3
non-shipstopper tracked items. Down from 23 in baseline sweep #1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 4 of the rust-orchestrator migration. Unifies the container lifecycle
surface behind a single trait so the RPC layer stops caring whether it is
talking to the dev or prod orchestrator.
* New trait core/archipelago/src/container/traits.rs: ContainerOrchestrator
with install / start / stop / restart / remove / upgrade / status / list /
logs / health, all keyed by app_id. Every method is async_trait-based.
* ProdContainerOrchestrator: the lifecycle methods are moved from inherent
impl into the trait impl (avoids name-shadowing recursion). Adoption and
reconcile remain inherent since only main.rs / BootReconciler call them.
* DevContainerOrchestrator: new trait impl that forwards to the existing
Dev-named methods, applying the dev container-name + port-offset rules
internally. New load_manifest_for() helper resolves app_id to
<data_dir>/apps/<app_id>/manifest.yml so trait-level install(app_id)
works in dev too. install_container(manifest, path) stays inherent for
the manifest-path RPC shape.
* RpcHandler now holds Option<Arc<dyn ContainerOrchestrator>> and, when in
dev mode, a separate Option<Arc<DevContainerOrchestrator>> for the
manifest_path install RPC. In prod mode RpcHandler::new() constructs a
ProdContainerOrchestrator and calls load_manifests() at startup.
* All seven container-* RPC guards no longer say dev mode required.
container-install still requires dev mode because its manifest_path
argument has no prod meaning; every other container RPC now works in both
modes via the trait.
BOOT STILL DOES NOT USE THIS. main.rs wire-up (Step 6) and BootReconciler
(Step 5) come next. Until then the prod orchestrator is constructed but nothing
populates /opt/archipelago/apps so it has zero manifests to manage, matching
the pre-Step-4 behaviour.
Verification: cargo build -p archipelago clean (11 expected unused method
warnings for methods not yet wired from main.rs). cargo test -p archipelago:
all 21 container::* tests pass (16 prod_orchestrator + 5 others). 24 other
test failures are pre-existing and unrelated (identity_manager / session /
wallet / mesh / credentials — all independently flaky on file-backed state).
Step 3 of the rust-orchestrator-migration. New file prod_orchestrator.rs (999 LOC)
implements the full public surface that will replace scripts/first-boot-containers.sh:
* install / start / stop / restart / remove / upgrade / status / list / logs / health
* adopt_existing: read-only scan that claims containers matching our manifests by
name, without recreating — preserves the v1.7.42 fixture on .116.
* reconcile_all: level-triggered, per-app failures collected rather than aborting.
* install_fresh: build-or-pull (Step 2 trait methods), relative build contexts
resolved against the manifest directory.
Naming rule (answered design Q1): UI app IDs (bitcoin-ui/electrs-ui/lnd-ui) get the
archy- prefix; backends keep their bare ID. An explicit extensions.container_name
always wins. Codified in compute_container_name() with unit tests for all three tiers.
Concurrency (answered design Q4): per-app tokio::sync::Mutex<()> created lazily,
protecting every mutating op against the reconciler loop. Acquiring the per-app
lock only needs a read lock on the map, so independent apps do not serialize.
16 tests: 3 sync naming rule tests + 13 tokio async tests covering install (pull,
build-absent, build-present, relative-context), reconcile (noop/exited/missing/
mixed-failure), adopt-by-name, upgrade sequence ordering, list filtering, health
state mapping, and unknown-app-id rejection. All pass.
Not wired into main.rs yet — that is Step 6. Crate builds clean with expected
unused warnings for the new re-exports.
Closes failure mode adjacent to FM3 (docs/bulletproof-containers.md): on
a syncing pruned node, bitcoind's RPC thread blocks for 5-10s during block
validation. The old 10s client-side timeout was rejecting roughly 30% of
UI calls even though the node was perfectly healthy. 20x stress test on
the live .116 node (caught in IBD catch-up at block 797k) used to drop
10 of 20 calls; now drops 0 of 20.
What changed:
- core/archipelago/src/api/rpc/bitcoin.rs: bitcoin_rpc_call now retries up
to 3 times with 500ms and 1500ms backoffs between attempts. Only
transient transport errors (timeout, connect refused, send/recv IO)
trigger retry. A well-formed bitcoind error response is surfaced
immediately - real RPC bugs are never masked.
- Per-attempt hard deadline (tokio::time::timeout, 15s) layered on top
of reqwest's own timeout, so DNS starvation or TLS wedging can't
steal the entire retry budget.
- handle_bitcoin_getinfo client builder gained a 3s connect_timeout
so a dead bitcoind is fast-failed inside the first attempt instead
of eating the whole 15s.
- Retry policy extracted into a RetryConfig struct so tests can dial
down timeouts to ~100ms per attempt. Production defaults live in
RetryConfig::production().
Not changed (tracked as follow-up):
- mesh/mod.rs bitcoin_rpc_getblockcount and related helpers use the
same 10s-timeout pattern. Not migrated to the new wrapper in this
release; scheduled for v1.7.43 alongside the render_bitcoin_conf
work.
- lnd/info.rs and electrs_status have similar 10s/15s timeouts but
different failure profiles - audit first, migrate only the ones
that actually exhibit the bug.
Tests: 6 new unit tests under api::rpc::bitcoin::tests, all passing.
Uses an in-process hyper server (already a transitive dep) to simulate
bitcoind responses; no new crates required.
- happy_path_first_attempt: no retry when first attempt succeeds
- retries_on_timeout_then_succeeds: first attempt times out, second
succeeds, returns OK (uses a short-timeout RetryConfig so the test
runs in <1s instead of 15s)
- retries_exhausted_on_persistent_connect_refused: all attempts fail
against a closed port, error bubbles up, elapsed time confirms
backoffs actually ran
- does_not_retry_on_rpc_level_error: bitcoind-returned error body is
surfaced immediately, no retry
- does_not_retry_parse_errors: non-JSON response (e.g. 503 with html
body) is NOT retried - guards against the tempting "retry all
non-2xx" mistake that would mask real bitcoind misconfig
- retry_budget_invariants: asserts total wall-time ceiling stays
under 60s so a bumped constant can't silently hang a UI call
forever
Validated live on .116: 20/20 bitcoin.getinfo calls succeed during IBD
catch-up (chain at block 797419 -> 797464), vs ~40% baseline under the
old 10s timeout. Worst-case latency was 48.9s during peak validation;
happy-path latency (cached result) remains 28-77ms.
Closes failure mode FM5 from docs/bulletproof-containers.md: the v1.7.38 +
v1.7.39 rollouts left every affected node on an unreachable UI (nginx 500)
with no recovery path short of SSH. This release adds a self-check
guardrail to the update flow.
What changed:
- apply_update() writes a pending-verify marker with old+new version and
a 150s deadline immediately before scheduling the service restart.
- verify_pending_update() runs from main.rs startup. If the marker is
present and within its freshness window, the new binary waits 15s for
nginx + backend to settle, then probes https://127.0.0.1/ every 5s for
up to 90s (self-signed certs accepted).
- On any probe success within the window, the marker is cleared and
nothing else happens.
- On window-exhaust, the new binary:
1. Moves the broken /opt/archipelago/web-ui to web-ui.failed.<ts>
(quarantined, not deleted, so we can post-mortem).
2. Restores web-ui.bak on top of web-ui.
3. Calls rollback_update() to restore the previous binary.
4. Updates state.current_version to reflect the rollback.
5. systemctl --no-block restart archipelago so the OLD binary boots.
- Markers older than 10 minutes are treated as stale and cleared without
probing, so a crashed-during-startup marker from weeks ago cannot
spontaneously roll back a healthy node on a later reboot.
- rollback_update() binary copy now goes through host_sudo instead of
tokio::fs::copy, so it escapes the service's ProtectSystem=strict
mount namespace. Without this, the rollback silently failed with
EROFS on /usr/local/bin and orphaned the rollback - the exact
opposite of what auto-rollback is for.
Tests: 4 new unit tests in update::tests covering marker round-trip,
absent-marker noop, no-panic on verify_pending_update with nothing to
verify, and an invariant assert that the 90s probe window stays below
the 600s stale threshold. All passing.
Side fix: scripts/create-release-manifest.sh was dying with exit 141
(SIGPIPE from tar tvzf pipe head pipe awk) under set -euo pipefail.
Replaced with a single awk NR==1 that doesn't short-circuit the upstream
pipe, so the release-build flow is idempotent again.
v1.7.38 and v1.7.39 both shipped with `./` inside the frontend tarball marked
drwx------ (700). Tar extraction preserves archive perms, so every node that
pulled the OTA landed with /opt/archipelago/web-ui at 700, nginx (www-data)
returned 500 "permission denied" on every page, and the browser showed
"Internal Server Error nginx". .116 hit this on both v1.7.38 and v1.7.39
rollouts. The v1.7.39 runtime self-heal in main.rs was the wrong layer —
systemd's ReadOnlyPaths namespace made /opt/archipelago read-only from inside
the archipelago service, so chmod from there returned EROFS.
Root cause: create-release-manifest.sh used mktemp -d (700 default umask) for
staging, then tar preserved that 700 in the archive's root entry.
Fix the archive itself:
- chmod 755 staging dir + `find -type d -exec chmod 755` + `-type f chmod 644`
before tar, so the on-disk entries are correct.
- tar --owner=0 --group=0 --mode='u=rwX,go=rX' to normalize archive perms
belt-and-braces in case file-mode drift ever reappears.
- Post-tar verify: `tar tvzf | head -1` must show drwxr-xr-x at root, or
the release script aborts before the manifest is even generated.
Binary unchanged semantically — the main.rs self-heal stays in as a last-
resort belt (can't hurt on nodes whose FS isn't namespace-isolated), and the
update.rs in-extractor chmod stays in so v1.7.40-onwards extractors are
double-safe. The authoritative fix is the archive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v1.7.38 shipped with an OTA bug: the tar-extracted staging dir inherited 700
perms and nginx (www-data) returned 500/403 on every request after the swap.
.116 hit this on rollout; had to chmod by hand to recover.
- update.rs: after extraction, explicitly chmod 755 dirs + 644 files on the
new staging dir before the mv into place, so nginx can stat/serve them.
- main.rs: self-heal on startup — if /opt/archipelago/web-ui is not
world-readable, run `sudo chmod -R u=rwX,go=rX` to repair. This is what
rescues nodes upgrading from v1.7.37/v1.7.38, since their extractor
(running on the old binary) doesn't have the chmod fix yet — the new
binary's first boot fixes the mess before nginx serves a single request.
Everything v1.7.38 shipped is still in this release:
- auth.rs auto-heals is_onboarding_complete() from setup_complete +
password_hash so nodes don't bounce back to /onboarding/intro after
browser clear / reboot / update
- useOnboarding tri-state: backend-unreachable no longer defaults to intro
- login sounds gated by isFirstInstallPhase() — silent after onboarding,
typing sounds unaffected
- FIPS app / Nostr Relay / Nostr VPN / Routstr / Penpot removed from
catalog + frontend + Rust + docker + icons; 15 image versions deleted
from tx1138, .168, gitea-local
- AIUI baked into release tarball via demo/aiui/
- prebuild hook syncs app-catalog/catalog.json → public/catalog.json
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Install flow
- api/rpc/package/install.rs: always append the literal image URL as a
last-resort pull candidate in do_pull_image, so images not carried by
any configured mirror (docker.io/bitcoin/bitcoin:28.4) still install
instead of masquerading as a generic pull failure across every mirror.
- api/rpc/package/install.rs: write_bitcoin_conf now skips on any stat
error, not just "file exists". Once bitcoin-knots' first-boot chowns
/var/lib/archipelago/bitcoin into the container's user namespace (700
perms, UID 100100/100101), the archipelago daemon can't even traverse
in — try_exists returns Err which unwrap_or(false) treated as "not
present" and drove a doomed write. Now errors out of the directory
traversal are treated as "conf already owned by container user" and
the write is skipped. Mirrors the lnd.conf pattern.
- api/rpc/package/install.rs: drop the hardcoded `prune=550` from the
conf default. Operators with multi-TB drives shouldn't be silently
pruned; users who want a pruned node can set it in bitcoin.conf
themselves. Full archive is the only honest default.
- api/rpc/package/config.rs: bitcoin-core now passes explicit
-server/-rpcbind/-rpcallowip/-rpcport/-printtoconsole/-datadir CLI
args. Vanilla bitcoin/bitcoin:28.4 has no entrypoint wrapper and
reads conf + argv only; without these the RPC listens on 127.0.0.1
inside the container and rootlessport can't reach it, so the
bitcoin-ui companion gets 502 on every /bitcoin-rpc/ call.
Bitcoin Knots keeps its own entrypoint-driven defaults.
- container/docker_packages.rs: split bitcoin-core out of the shared
AppMetadata arm. bitcoin-core now surfaces as "Bitcoin Core" with
bitcoin-core.svg and a Reference-implementation description; the
bitcoin + bitcoin-knots ids keep the Knots branding. Fixes the home
card showing "Bitcoin Knots" for a Core install.
Bitcoin node UI (docker/bitcoin-ui)
- index.html: impl name/tagline/logo now dynamic. applyImplBranding()
reads subversion from getnetworkinfo — /Satoshi:X/Knots:Y/ resolves
to Bitcoin Knots, plain /Satoshi:X/ resolves to Bitcoin Core. Both
get their own icon and subtitle. Settings modal replaced its
hardcoded Regtest/txindex=1/port-18443 placeholders with live values
from getblockchaininfo + getindexinfo + getzmqnotifications.
- index.html: new Storage info card (Full Archive · X GB /
Pruned · X GB from blockchainInfo.pruned + size_on_disk) visible on
the main dashboard, same level as Network. Settings modal mirrors it
with the prune height when applicable.
- Dockerfile + assets/: bitcoin-core.svg, bitcoin-knots.webp, and the
bg-network.jpg used by the dashboard are now COPY'd into the image
under /usr/share/nginx/html/assets. Previously the <img src> pointed
at paths that 404'd into the SPA fallback and the onerror handler
hid the broken logo silently.
Frontend
- appSession/appSessionConfig.ts: add bitcoin-core to APP_PORTS (8334),
HTTPS_PROXY_PATHS (/app/bitcoin-ui/), and APP_TITLES (Bitcoin Core).
Without these the AppSessionFrame showed "No URL found for
bitcoin-core" and the home/app-list title fell through to the raw id.
- settings/AccountInfoSection.vue: backfill What's New entries for
v1.7.31 through v1.7.37 that had been missed in earlier cuts.
Release plumbing
- releases/v1.7.37-alpha/: binary + frontend tarball.
- releases/manifest.json: v1.7.37-alpha, sha256/size refreshed.
- Cargo.toml / package.json: version bumps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- neode-ui/public/assets/img/app-icons/bitcoin-core.svg (NEW): 256×256
Umbrel community Bitcoin icon sourced from getumbrel.github.io/
umbrel-apps-gallery/bitcoin/icon.svg. Referenced by the static
catalog, the curated fallback, and the upstream lfg2025/app-catalog
entry so every surface shows the same image.
- app-catalog/catalog.json + neode-ui/public/catalog.json: add
bitcoin-core (v28.4) entry pointing at bitcoin/bitcoin:28.4. Same
entry pushed to the lfg2025/app-catalog repo on .160 and the local
gitea mirror so nodes see it without needing a full archipelago
update. Sovereignty Stack entry added to FEATURED_DEFINITIONS with
a description that frames it as a Knots alternative, not a rival.
- core/archipelago/src/api/handler/mod.rs: handle_app_catalog_proxy
is now instance-scoped (&self) and derives its upstream list from
load_registries — each active container registry contributes one
`<scheme>://<reg.url>/app-catalog/raw/branch/main/catalog.json` URL
in priority order (scheme follows tls_verify). When the operator
switches mirrors in Settings, the App Store now follows. Falls back
to the legacy hardcoded .160/tx1138 pair only when registry config
can't be loaded, so the App Store still renders on nodes that
haven't persisted one yet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- core/archipelago/src/bootstrap.rs (NEW): embed scripts/container-doctor.sh
and image-recipe/configs/archipelago-doctor.{service,timer} via
include_str! and sync to disk + enable the timer on every archipelago
startup. Idempotent (content-hash compare), dev-box symlink guard keeps
the git checkout untouched, best-effort (warn-only on failure) so
bootstrap never blocks server readiness. Wired in main.rs as a
background tokio task.
- scripts/container-doctor.sh: add fix_rootless_netns_egress(). Detects
when the rootless-netns has lost its pasta tap (container-to-container
still works but outbound DNS/TCP fails) via an nsenter probe into
aardvark-dns; with a two-probe 10s debounce to rule out transients and
a host-precheck that bails out if the host itself is offline. When the
rootless-netns is truly broken, does a graceful podman stop --all /
start --all so pasta + aardvark-dns rebuild the netns from scratch.
Bitcoin-knots and every other outbound container recover in one cycle.
- core/archipelago/src/update.rs: host_sudo → pub(crate) so bootstrap.rs
can reuse the existing systemd-run escape hatch.
- apps/bitcoin-core/manifest.yml: bump app version 24.0.0 → 28.4.0 and
image bitcoin/bitcoin:24.0 → bitcoin/bitcoin:28.4. Resources aligned
with the real container-specs.sh large-disk tune (4 GiB memory cap,
cpu_limit: 0 so bitcoind can run -par=auto across every core).
- neode-ui/src/views/apps/AppCard.vue + Apps.vue: add an Update button
+ Updating spinner to every app card that has available-update set.
Wires through serverStore.updatePackage(id) — the same RPC the detail
view already calls. common.update / common.updating i18n keys added in
en.json and es.json.
- core/archipelago/src/identity_manager.rs: add create_from_signing_key()
that mirrors an existing Ed25519 key as a manager-level identity with
a deterministic id (`node-<pubkey16>`). Idempotent across restarts,
gets the hex-SVG master avatar.
- core/archipelago/src/server.rs: the auto-create path on first boot now
mirrors the node's own signing_key (seed-derived on onboarded installs)
as a "Node" identity instead of generating a random "Default" keypair.
Once this ships, the DID on the Web5 DID Status card (via node.did
RPC), the Node entry on the Identities page (via identity.list), and
the DID used for peer-to-peer connects (via server_info.pubkey) all
resolve to the same seed-derived pubkey.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- useOnboarding.ts: when the backend gives a definitive answer
(true/false, not a null retry failure), re-seed the
neode_onboarding_complete localStorage flag accordingly. Fixes the
case where a user clears site data on an already-onboarded node —
OnboardingWrapper's useVideoBackground computed reads localStorage
synchronously, so without this re-seed the intro video would fire
again on /login even though RootRedirect correctly sent them
straight to /login.
- OnboardingWrapper.vue: login background now rotates through
bg-intro-1..6 on each /login mount, with the current index
persisted to localStorage (neode_login_bg_idx) so subsequent
logouts advance rather than repeat the same image.
- Dashboard.vue: subsequent-login branch drops the 1.2s showZoomIn
entirely. Only the first dashboard entry after onboarding plays
the full zoom + glitch reveal; every re-login now just fades in
with the welcome typing (~300ms).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- useOnboarding.ts: prefer the backend over localStorage when checking
onboarding completion. The old order (localStorage first) meant any
browser that had ever onboarded a node would treat every new fresh
node as already-onboarded and skip the wizard, dumping the user
straight at the inline set-password form. Backend is now authoritative;
localStorage stays as the offline fallback.
- OnboardingWrapper.vue: skip the intro video on `/login` once
`neode_onboarding_complete` is set. Returning logged-out users now
get the static lock-screen background + glitch overlay instead of
replaying the full intro on every logout.
- RootRedirect.vue: when the health check fails, only show the full
BootScreen if the node was never onboarded. For already-onboarded
nodes (i.e. an OTA-update blip), keep the spinner and poll the
health endpoint every 2s for up to 60s before falling back to the
boot screen. Fixes the "fake boot loader" / "server starting up"
screens flashing on every successful update.
- loginTransition store: new `justCompletedOnboarding` flag distinct
from `justLoggedIn`. Set true only by the inline setup-password
flow (handleSetup). Dashboard.vue branches on it: full glitch+zoom
reveal for the post-onboarding entry, quick zoom + welcome typing
on every other login (no triple glitch flashes, ~1.2s vs 8s).
- vite.config.ts: bump assets cache from `assets-cache-v2` to
`assets-cache-v3` so service workers running the previous bundle
invalidate their cache and pick up the new UI cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- HOTFIX: v1.7.31-alpha's frontend tarball was packaged with a
`neode-ui/` top-level directory instead of the flat layout v1.7.30
and earlier used. Nodes that applied v1.7.31 ended up with
`/opt/archipelago/web-ui/neode-ui/index.html` instead of
`/opt/archipelago/web-ui/index.html`, and nginx returned 403/500.
v1.7.32's tarball is built with `tar -C web/dist/neode-ui .` so
files land directly at web-ui root. Broken nodes auto-heal on this
update (web-ui dir is replaced).
- transport/lan.rs: add Drop impl that calls ServiceDaemon::shutdown()
on the mdns_sd daemon. Without this the OS thread it spawns, plus
the blocking `receiver.recv()` task, keep the tokio runtime alive
past SIGTERM — long enough for systemd's TimeoutStopSec to SIGKILL
the service and mark it Failed. Was visible on every update:
"shut down cleanly" logged, then 15s later systemd forcibly kills.
- main.rs: after logging "Archipelago shut down cleanly", call
`std::process::exit(0)` explicitly. Belt-and-suspenders against
any future non-daemon thread creeping in (reqwest resolver pool,
etc.) and causing the same SIGKILL regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Backend: install.rs registry reachability probe now strips the
`host[:port]/namespace` suffix before appending `/v2/` (the Docker
V2 API lives at the host root, not under the namespace) and accepts
HTTP 405 in addition to 200/401 as "registry daemon alive". This
fixes false "unreachable" reports on the Test button for Gitea and
other registries that protect their /v2/ endpoint.
- Backend: stacks.rs install_indeedhub_stack now force-removes any
leftover indeedhub-* containers and indeedhub-net before creating
the stack. A partial install (or the old first-boot stub racing the
installer) used to leave containers around that blocked re-install
with "name already in use". Re-running the App Store install now
self-heals.
- Backend: registry.rs load_registries auto-merges any default
registry URLs missing from the saved config (appended with priority
max+10+i, persisted). Lets new default mirrors (e.g. Server 3 OVH)
roll out to existing nodes without manual config edits. Explicit
removals still stick — URLs absent from disk AND absent from
defaults stay gone.
- Backend: update.rs adds DEFAULT_TERTIARY_MIRROR_URL at
http://146.59.87.168:3000/ (Server 3 OVH) to default_mirrors, with
the same auto-merge-on-load behavior as registries. Test updated
for 3-mirror default (.160, tx1138, .168).
- Scripts: dropped the first-boot IndeedHub stub (~38 lines in
first-boot-containers.sh §8b). It predated the proper stack
installer, raced it, and was the main source of the name-conflict
mess the stacks.rs cleanup above now also guards against.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Backend: unified pull-progress streaming across primary AND fallback
registries. Earlier code only streamed for the primary attempt; if it
failed fast (VPS 404, etc.) the UI froze at 0% until the fallback
finished. The waterfall now uses a single shared helper that streams
podman stderr through update_install_progress for every URL tried.
- Backend: PackageDataEntry gains uninstall_stage, set at each phase of
handle_package_uninstall ("Stopping containers (i/total)",
"Cleaning up volumes", "Removing app data"). State flips to Removing
during the pipeline.
- Frontend: MarketplaceAppCard renders the live progress bar with byte
counts during installs, matching the System Update download bar style.
- Frontend: AppCard renders the live uninstall stage label per app.
Modal closes immediately on confirm so concurrent uninstalls each
show their own progress on their own card.
- Cleanup: removed dead helpers (image_candidates, rewrite_for_primary,
primary_image_url, pull_from_registries_with_skip) made unused by
the install.rs refactor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New Settings → App registries page (/dashboard/settings/registries)
that mirrors the update-mirrors experience: list of configured
registries, test reachability, set primary, add/remove. New
registry.set-primary RPC; existing registry.{list,add,remove,test}
reused.
- Default RegistryConfig flipped: VPS (23.182.128.160:3000/lfg2025) is
now Server 1 (primary), tx1138 is Server 2 (fallback).
- Install pipeline now rewrites the first pull to the primary registry
URL before attempting it. Before this, installs always hit whichever
registry the image was hardcoded to, so changing the primary didn't
actually affect where images came from. On failure, the existing
fallback walk skips the primary (already tried) and walks the rest.
- App catalog proxy UPSTREAMS order flipped so the catalog follows the
same VPS-first rule.
- Reboot overlay: animated "a" logo now sits in the center of the ring
(matches the screensaver composition). Extracted the logo-wrapper
pattern inline.
7/7 registry tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New reboot progress overlay: full-screen black with the screensaver's
pulsing ring, rebooting → reconnecting → back-online → stalled stages,
elapsed counter, auto-reload on health-check success, manual reload
button at 3 min stall. Mirrors the existing update overlay.
- Ring extracted from Screensaver.vue into a reusable ScreensaverRing
component so the reboot overlay reuses the same animation.
- default_mirrors() now puts the VPS as Server 1 (primary) and tx1138 as
Server 2 — new nodes fetch manifests from VPS first; existing nodes
keep whatever mirror order they've customized.
- What's New entry prepended for v1.7.28-alpha.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New "Served by {mirror}" line on the System Update page so operators can see
which mirror actually served the available manifest (vs. which is configured
primary). Backend threads the served URL through UpdateState.manifest_mirror.
- New update.test-mirror RPC + per-row lightning-bolt button that pings a
mirror and renders reachable/latency or error inline under the URL.
- UI polish on the mirrors section: Set Primary, Remove, and the new Test
action are compact icon buttons; add-mirror form moved into a dialog.
- "What's New" block prepended for v1.7.27-alpha.
21/21 update module tests pass. vue-tsc + vite build clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a multi-mirror manifest fetch. `check_for_updates` walks a
configurable list (data_dir/update-mirrors.json) in priority order
and falls through to the next mirror on any HTTP / parse / timeout
failure. Two defaults bake in: Server 1 (git.tx1138.com) and Server 2
(23.182.128.160:3000).
Critical fix: after parsing a manifest, rewrite every component's
`download_url` so its origin matches the manifest URL we fetched.
Before this, the manifest hard-coded absolute URLs pointing at one
specific server — so even when a node fetched the manifest from a
faster mirror, the actual 200MB download went back to the slow
original. Now the faster mirror wins end-to-end.
New RPCs: update.list-mirrors, update.add-mirror, update.remove-mirror,
update.set-primary-mirror. New UI section on the System Update page
for operator management. 5 new unit tests for origin parsing and
manifest rewriting (21/21 green).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-adds the TCP transport (`0.0.0.0:8443`) to the rendered fips.yaml
alongside UDP. Upstream factory default enables both; we had
inadvertently narrowed to UDP-only when the yaml rewriter was last
touched, which left nodes unable to reach fips.v0l.io (the public
anchor only answers on TCP right now) or talk across networks that
block UDP.
Backend startup now compares the installed yaml against the current
rendered schema and restarts whichever fips unit is active when they
differ — so OTA-upgrading nodes pick up the new transport without
anyone having to click Reconnect.
Dropped the earlier plan to auto-add federated peers as seed anchors:
invites don't carry a FIPS-reachable IP:port, and once TCP reconnects
the public mesh, federated peers become npub-routable without needing
a seed entry.
Seed Anchors modal cleanup: replaced malformed header icon with a
three-arc broadcast glyph, and the close button now matches the
What's New modal (embedded in the card header, same icon + hover
style) instead of the earlier floating off-design placeholder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The npm run build step in the release ritual had been silently failing
for roughly seven releases. vue-tsc died with EACCES on a root-owned
node_modules/.tmp, exited non-zero, and my `tail -5` of the build
output happened to only show vite's precache summary — which makes
vite look successful even when the typecheck that precedes it failed.
The resulting archipelago-frontend-*.tar.gz files were rebuilds from
whatever content happened to live in web/dist/neode-ui/ at the moment
(files left over from v1.7.9, owned root:root from an earlier sudo'd
operation, unchanged since).
Fixed by chowning both paths back to the archipelago user and
rebuilding. Every published frontend tarball from v1.7.17 through
v1.7.23 therefore shipped the same frozen UI; v1.7.24 is the first
release in that stretch whose frontend actually matches its backend.
Recorded the build-verification rule as a persistent feedback memory
(feedback_frontend_build_verify.md) — future ships must grep the
packaged tarball for the new version string before push.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a gear button next to the FIPS Mesh card's status pill that
opens a Teleport-ed modal containing FipsSeedAnchorsCard. The card
was landed on disk in v1.7.21 but never wired into a UI entry point
per the entry-point convention, so users couldn't access the
Add/Remove/Apply controls at all. One gear click now opens them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- fips::service::active_unit() picks whichever fips unit is running
(archipelago-fips.service vs upstream fips.service) so
handle_fips_restart and handle_fips_reconnect don't silently no-op
on hosts where the archipelago-managed unit was never created.
- peer_connectivity_summary(anchor_candidates) replaces the old
identity-cache check. anchor_connected is now true when at least
one authenticated peer's npub matches the public anchor OR any
entry in seed-anchors.json, which matches what the user actually
cares about ("am I in the mesh?") rather than what the card used
to claim ("is this one specific public anchor reachable?").
- FipsStatus::query takes data_dir now (so it can read seed-anchors)
rather than identity_dir. All call-sites updated.
- handle_fips_reconnect re-pushes seed anchors after restart so the
new daemon gets dialed without waiting for the 5-min apply loop.
- FipsNetworkCard label drops "(fips.v0l.io)" — misleading now that
multiple anchors may be configured.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a local seed-anchor list at <data_dir>/seed-anchors.json. Each
entry is {npub, address, transport, label}. On archipelago startup
and every 5 minutes the list is pushed into the running fips daemon
via `fipsctl connect <npub> <addr> <transport>`, so a cluster can
anchor itself independently of the global fips.v0l.io. A flaky or
unreachable public anchor no longer strands a fresh install.
New RPCs:
- fips.list-seed-anchors
- fips.add-seed-anchor (validates npub1… + host:port)
- fips.remove-seed-anchor
- fips.apply-seed-anchors (on-demand re-dial)
New standalone UI card at views/server/FipsSeedAnchorsCard.vue. Not
wired into Home.vue / Server.vue — operator places it per the
entry-point convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 3AM auto-update path called std::process::exit(0) immediately
after apply_update returned. apply_update had already spawned a 2s-
delayed systemctl restart, but exit(0) killed the runtime before that
spawned task could run — and the unit's Restart=on-failure does not
trigger on a clean exit 0, so the service stayed dead until someone
SSH'd in and started it manually (.253 hit this today).
Scheduler now returns from the task without killing the process;
apply_update's existing restart path (same one the UI's Install
Update button uses) brings the new version up cleanly.
Also hardens the ISO CI: the AIUI inclusion step now falls back to
extracting from the newest release tarball if the runner's cached
/opt/archipelago/web-ui/aiui path is missing, so a reprovisioned
runner can't silently ship a frontend tarball without AIUI. The ISO
build step also sanity-checks the binary exists before invoking the
builder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
load_state now drops any stored available_update whenever the running
binary version differs from what's on disk — the old migration only
cleared it when the stale entry happened to match the new version, so
skipping releases (e.g. sideloading 1.7.16 → 1.7.18 without 1.7.17)
left a pointer to an intermediate version as the "update available",
which the UI then offered as a downgrade prompt.
check_for_updates also uses a numeric version comparator so a stale or
cached manifest with an older version can't offer itself as an
update, and 1.7.10 correctly outranks 1.7.9 past the single-digit
patch boundary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flip transitively-discovered federation peers to Trusted instead of
Observer. Hints are already only ingested from peers we trust and only
peers we trust are re-exported via build_local_state, so the chain of
trust is already vetted end-to-end — making the user promote each
newcomer by hand was friction with no security win.
Backend:
- federation/sync.rs: merge_transitive_peers now inserts TrustLevel::Trusted
(doc comment updated to explain the transitive-trust rationale)
- update.rs: info! log at download start (version, components, total_bytes,
staging path), cancel (staging wiped?, marker cleared?), and apply (backup
path) so journalctl reveals where a stuck update actually is
Frontend:
- SystemUpdate What's New block gets a v1.7.18-alpha entry
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Federation join flow now notifies the inviter with the joiner's name and
immediately bumps state so the Federation UI reloads without a manual
Sync click. Accepting an invite that points back at the local node is
rejected up front (DID/pubkey/onion match). After a peer joins, we spawn
a transitive sync that pulls the new peer's federated peer hints so all
nodes in the federation learn about each other as Observer entries.
Federation.vue polls every 5s while mounted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
download_update
Each component download is now resumable via HTTP Range requests
(Range: bytes=N-) and retried up to 6 times with exponential
backoff (5/15/30/60/120/180s). On a dropped connection the next
attempt picks up at the last written byte offset instead of
restarting at zero. Streams via reqwest::Response::chunk() to the
staging file so a 160 MB frontend tarball doesn't sit in RAM. SHA
is verified over the complete file at the end of each component;
mismatch nukes the staged file and restarts from scratch.
Real download progress counters
New AtomicU64 globals DOWNLOAD_BYTES/DOWNLOAD_TOTAL are updated
from the chunk loop. update.status exposes them as
download_progress.{bytes_downloaded, total_bytes, active}. The
SystemUpdate.vue progress bar now polls update.status every
second instead of incrementing a fake random counter — and
crucially, if the user navigates away and back, the component
picks up the in-progress download from the backend atomics
immediately.
Update-check retries
handle_update_check now retries the manifest fetch up to 3 times
with a 5s gap if the first try hits a transport error, so a
momentary gitea hiccup doesn't make a node report "up to date"
when there actually is a new release. Tight 10s connect timeout
per attempt keeps the total bounded.
Artefacts:
archipelago 1070c87f…c081c162b 40584792
archipelago-frontend-1.7.15-alpha.tar.gz 8e630eba…63fd43f 162078068
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Install UX
SystemUpdate.vue now shows a full-screen overlay after apply: the
BitcoinFaceAscii logo, a target-version label, an indeterminate
progress stripe (solid orange; solid green on ready), and an
elapsed-time readout. Polls /health every 1.5s and auto-reloads
once the backend reports the new version. 3-min stall → "Reload
now" button. Download UI also shows a spinner + "Finishing
download — verifying checksum…" while the fake bar sits at 95%.
FIPS reconnect — for real this time
New fips.reconnect RPC does stop → start → wait 20s → re-poll →
classify. Classification buckets: connected / daemon_down /
no_seed_key / no_outbound_udp_or_anchor_down / peers_but_no_anchor,
each with a plain-language hint surfaced verbatim by the Reconnect
button. The real reason nodes like .198/.253 couldn't reach the
anchor: identity::write_fips_key_from_seed was writing fips_key.pub
as a bech32 npub TEXT file, but upstream fips expects 32 raw
bytes. The daemon silently authenticated with garbage. Fix:
PublicKey::to_bytes() → raw 32 bytes, and new
fips::config::normalize_pub_file migrates legacy files by decoding
the npub and rewriting in place. fips.reconnect also re-installs
the config + healed keys to /etc/fips before restarting.
AIUI preservation + restore
apply_update was wiping /opt/archipelago/web-ui/aiui because the
Vue build doesn't include it — every OTA lost the Claude sidebar.
The preserve block now copies aiui/ + archipelago-companion.apk
from the old web-ui into the staging dir before the swap, and
prefers new-tar versions if present. To restore it on the three
nodes that already lost it (.116/.198/.253), this release bundles
the 85 MB aiui build into the frontend tarball. Frontend component
size is now ~155 MB.
Download / install timeouts
Backend download client timeout 1800s → 3600s (1 h). Larger
tarball + slow gitea raw throughput put us above the old cap.
Frontend update.download rpc timeout 30 min → 65 min to match.
package.install rpc timeout 15 min → 45 min — IndeedHub pulls
6 images and was timing out mid-install.
UI nit
"Rollback to Previous" → "Rollback Available".
App-catalog proxy already landed in v1.7.13.
Artefacts:
archipelago 725e18e6…3c525e6 40462288
archipelago-frontend-1.7.14-alpha.tar.gz c35284be…ff2c16 162077052 (+aiui)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Discover / Marketplace page fetched the app catalog directly from
git.tx1138.com/lfg2025/app-catalog/raw/.../catalog.json in the
browser. Two blockers hit the fleet simultaneously: (1) tx1138's
Gitea doesn't emit Access-Control-Allow-Origin so the HTTPS fetch
got CORS-blocked; (2) the HTTP IP-port fallback
(http://23.182.128.160:3000/...) falls outside the node's
`connect-src` CSP. Users saw the hardcoded fallback instead of the
live catalog.
Backend: new authenticated GET /api/app-catalog handler uses reqwest
to pull catalog.json server-side (15s timeout) and returns it with
application/json + 1h Cache-Control. Tries the HTTPS URL first,
HTTP IP-port second.
Frontend: curatedApps.ts now calls /api/app-catalog (same-origin,
no CORS/CSP) with credentials included so the session cookie
authenticates the proxy. Baked /catalog.json stays as the last
resort.
Artefacts:
archipelago 0aaf7262…b979f22c 40371192
archipelago-frontend-1.7.13-alpha.tar.gz 27505811…efc6f4142 76982505
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Version-only bump. Sits above v1.7.11-alpha which user has verified
runs the full Install Update pipeline end-to-end (check → download
→ install → auto-restart). Freshly-installed nodes from the 1.7.11
ISO will see 1.7.12 as their first OTA target.
Frontend tarball byte-identical to v1.7.11 (same sha).
Artefacts:
archipelago 247f65c2…54f40df9 40385472
archipelago-frontend-1.7.12-alpha.tar.gz 0644a436…54f58 76983846 (reused)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Version-only bump. Frontend tarball byte-identical to v1.7.10. First
OTA-testable release where the running backend (v1.7.10) has the
host_sudo/systemd-run apply fix — clicking Install Update should
walk through check → download → install → auto-restart with no
manual intervention.
Artefacts:
archipelago cf003f62…65465f 40378752
archipelago-frontend-1.7.11-alpha.tar.gz 0644a436…54f58 76983846 (reused)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
THE apply fix
archipelago.service uses ProtectSystem=strict, so /opt and /usr are
read-only inside the service's mount namespace. sudo inherits that
namespace — every sudo mkdir/mv/chown from apply_update was hitting
EROFS even as root. Every prior "Failed to apply update" was a
symptom of this. New `host_sudo()` helper wraps every filesystem
call in `sudo systemd-run --wait --collect --pipe -- <cmd>`, which
spawns a transient unit with systemd's default (no ProtectSystem)
protections — the command runs in the host namespace and can touch
/opt/archipelago + /usr/local/bin normally.
FIPS cascade (#2)
Home.vue and Server.vue both carry a FIPS row that previously only
looked at {installed, service_active, key_present}. Now they also
read anchor_connected + authenticated_peer_count and mirror the
full FIPS card: green "Active · N peers" when healthy, orange "No
anchor" when the DHT bootstrap has failed.
Profile paste URL fallback (#4)
Web5Identities.vue list + editor previously had `@error="display:none"`
on the <img>, which hid the tag without re-rendering the fallback —
a broken pasted URL showed up blank. Replaced with reactive
pictureLoadFailed / listPictureFailed flags plus a watcher that
resets on URL change. Broken URL now falls back to the initial (or
identicon for seed-derived identities).
Small-upload data URL (#3)
Uploaded profile pictures ≤ 64 KB are now inlined as
`data:image/png;base64,...` into profile.picture on the client
before calling update-profile. That kind-0 event is fetchable by
any Nostr client — no Tor needed. Larger uploads fall back to the
onion-rooted public_url with a hint telling the user to paste a
public https:// URL for broader visibility.
Deferred: #1 FIPS Reconnect "actually fixes" — the current Reconnect
calls fips.restart which clears the daemon state, but when the
anchor is truly unreachable (UDP 8668 blocked by network/ISP), no
amount of restart can help. A richer diagnostic is out of scope for
this bundle.
Artefacts:
archipelago 4a77c704…82aa6f8 40379696
archipelago-frontend-1.7.10-alpha.tar.gz 0644a436…54f58 76983846
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Version-only bump. First release where .116/.198/.253 (running v1.7.8
with the mv-based apply) should walk through Check → Download →
Install → auto-restart cleanly via UI, no sideload intervention.
Artefacts:
archipelago 1ec7383d…301629 40378536
archipelago-frontend-1.7.9-alpha.tar.gz 4fb79664…0172e9 76984615 (reused)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
apply_update's binary swap called `sudo install -m 0755 src
/usr/local/bin/archipelago`. install opens the destination for write
with O_TRUNC; the kernel returns ETXTBSY (exit 1) when the path is a
currently-running executable, which it always is during apply because
apply_update is called by the archipelago RPC handler — running as
archipelago itself. Every previous "Failed to apply update" was this
one root cause; the manual sideload path only worked because we
stopped the service first.
rename() doesn't modify the file it replaces — it repoints the path
at a new inode while the old inode stays alive for any process that
has it mapped. `mv` uses rename(). Switched to `sudo mv` (with prior
chmod+chown on the staging file) so the swap is atomic and tolerant
of the running binary.
Frontend tarball byte-identical to v1.7.7-alpha; only the binary
version string changes.
Artefacts:
archipelago 2753daec…48094d 40377648
archipelago-frontend-1.7.8-alpha.tar.gz 4fb79664…0172e9 76984615 (reused)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure version bump. No code changes. First release shipped with the
reinforced apply_update (timestamped staging + all-mv) and frontend
with 95% progress cap — this OTA should walk through cleanly from
.116/.198/.253 without any sideload intervention.
Artefacts:
archipelago e3f1740d…006025 40373392
archipelago-frontend-1.7.7-alpha.tar.gz 4fb79664…0172e9 76984615 (reused)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
