package.stop/restart on orchestrator-managed stacks (immich, indeedhub,
btcpay, netbird, mempool) addressed live CONTAINER names; unknown app ids
fell through to raw podman stop/start, racing systemd's --rm cleanup of the
quadlet unit — the container vanished on stop and the start half found
nothing (immich RESTART FAIL + indeedhub stuck restarting, gate 2026-07-09).
Now stack ops resolve member APP ids (reverse start order for stop) so the
orchestrator drives the quadlet .service; user-stop markers are written for
member app ids too (covers absent containers), and the orchestrator stop
fan-out treats an already-missing container as stopped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/var/log/archipelago/container-installs.log has been 0 bytes since April:
the service sandbox (ProtectSystem=strict class) leaves /var/log
read-only for the unit, the open() fails, and install_log() drops every
line by design (fire-and-forget). These START/STOP/RESTART/FAIL
breadcrumbs are the primary forensic trail for gate failures, so mirror
every line to tracing (journald) unconditionally and keep the file
append as best-effort for hosts where the path is writable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One legacy app per gate run stuck at 'stopping'/'starting' past the test
window while its container was already settled (vaultwarden:stop run C,
jellyfin:stop run D, uptime-kuma:start run E — .228, 2026-07-09). The
package scanner already sees the truth every 60s but
merge_preserving_transitional refused to report it until the RPC worker
wrote the final state, and the workers can legitimately trail the
container by minutes:
- stop workers queue behind the orchestrator per-app lock, which the
reconcile host-port repair path holds through multi-minute stability
waits (repair_manifest_host_ports_after_stability: 5s + 5s probe +
restart + 60-420s port wait + 15-90s stable-running);
- start workers hold Starting through the full readiness wait —
host_port_wait_timeout_secs is 420s for uptime-kuma (HTTP probe),
longer than any UI/test patience — and the Starting stuck-timeout is
the 20-minute INSTALLING one.
New merge rules, both truth-driven by podman's live view:
- (Stopping, Stopped) + user-stop marker → Stopped: the scanner only
normalizes exited→Stopped for user-stopped apps, so this is exactly
"the user asked for a stop and the container has exited".
- (Starting, Running) → Running: the start visibly succeeded; live
health readings are merged separately and keep reporting readiness.
Restarting is deliberately NOT resolved by a Running scan — mid-restart
readings are the pre-stop container.
Tests: merge_tests 13/13 (4 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.start/stop/restart reply immediately and run their multi-container
sequences in spawned tasks with no coordination. Back-to-back RPCs on the
same app interleave those sequences: gate run D (.228, 2026-07-09 04:28)
fired stop→start→restart on mempool; the start's member bring-up raced the
still-running stop's member shutdown — archy-mempool-db was stopped 2s
after it started and the stack finished with ZERO containers (tests 95/123/
124). A user double-clicking restart in the UI can do the same.
Workers now take a per-app tokio Mutex (fair/FIFO, keyed by the normalized
orchestrator app id) as their first await, so queued ops run in RPC arrival
order and the final state matches the last request. The RPC reply stays
immediate.
Known limit: container lists are still computed at RPC time (pre-lock), so
a stop landing mid-start can still return 'No containers found' — it can no
longer interleave/corrupt, only error. Reconciler/health-monitor restarts
don't take the lock yet (separate paths, same candidate follow-up).
Also from run D, filed separately: jellyfin/vaultwarden (legacy containers
on a quadlet node) stop fine but the package state sticks at 'stopping' —
'Failed to stop jellyfin.service: Unit not loaded' error path suspected.
Tests: runtime 9/9; archipelago builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second instance of the stop→start destruction class from tonight's gate
runs (.228): package.stop of a multi-container stack marks every member
user-stopped and quadlet-removes their containers; package.start's fallback
then resurrected only the bare package id, leaving the other members absent
AND user-stopped — indeedhub lost all 7 containers this way (mempool needed
its umbrella alias for the same reason).
- New stack_member_app_ids map (real manifest app ids, dependency order) —
used by the start fallback when a stack has no live containers, and by
package.restart instead of hard-failing "No containers found".
- orchestrator_uninstall_app_ids grows the missing indeedhub arm (same
reconciler-ghost rationale as the immich arm).
- do_orchestrator_package_start no longer aborts the whole sequence when one
member's unknown-app-id fallback fails (the #73/#74 wish) — collects and
continues.
NOTE: extends the known §G per-app-map debt (tracker updated) — the proper
home for stack membership is a manifest field; parked post-tag by design.
Tests: package:: 48/48 incl. new fallback + indeedhub-uninstall coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the .228 lifecycle gate (2026-07-08, first quadlet-mode run):
stop→start on the legacy `mempool` umbrella app id destroyed the mempool
deployment. Under quadlet, package.stop removes the containers; package.start
then failed with `unknown app_id: mempool` because load_manifests drops the
umbrella manifest whenever the three split-stack members are loaded — leaving
nothing to recreate from and the stack down.
Two fixes:
- prod_orchestrator start/stop/restart now resolve the historical
`mempool`/`mempool-web` id to the split members (archy-mempool-db,
mempool-api, archy-mempool-web) whenever the umbrella manifest was dropped —
the same alias install already implements ("installing mempool assembles the
split stack"). Restart falls back to start for members whose container/unit
is gone. Legacy umbrella-only nodes are unaffected (alias inactive while the
umbrella manifest is live).
- is_missing_container_error (3 sites) now recognizes podman 5.x's
`no such object` inspect phrasing, so a genuinely absent container is
classified as missing instead of surfacing as a hard inspect error.
Tests: 2 new alias tests + 2 classifier tests; prod_orchestrator suite 61/61
green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second false-drift source found live on .228 right after the entrypoint
split fix: a YAML `>-` script with more-indented continuation lines
keeps literal newlines in custom_args, but shell_join flattens them to
spaces when writing the quadlet Exec= line, so Config.Cmd stores spaces
where the manifest has newlines. bitcoin-knots and fedimint-gateway
read as permanently command-drifted over line breaks alone (only the
restart-sensitive guard kept them from recreate-looping like electrumx).
Normalize both sides the way shell_join does before comparing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
container_command_drifted compared the manifest's entrypoint and
custom_args against Config.Entrypoint and Config.Cmd separately, but
quadlet's Entrypoint= takes a single value, so a manifest entrypoint of
[sh, -lc] is written as Entrypoint=sh + Exec=-lc ... and podman reports
entry=[sh], cmd=[-lc, script]. Same argv, different split — every
quadlet-created app with a multi-element entrypoint read as drifted
forever. On .228 this recreated electrumx on every reconcile pass (114
times in 6h); bitcoin-knots and fedimint-gateway showed the same false
drift and were only spared by the restart-sensitive guard.
Compare the concatenated argv (entrypoint ++ args vs Entrypoint ++ Cmd)
instead — that is what actually runs. When the manifest declares no
custom_args, only the entrypoint prefix must match, since the trailing
Cmd may be the image's baked-in CMD.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The packaged reticulum-daemon is a PyInstaller one-file binary: the
bootloader extracts a ~48M _MEI* dir into TMPDIR and only deletes it on
clean exit. ReticulumLink's shutdown SIGKILLed the process group
immediately (kill_on_drop + instant SIGTERM+SIGKILL in Drop), so the
cleanup never ran and every reconnect/restart stranded another 48M in
/tmp — filling .116's 12G tmpfs in ~4h.
Two-pronged fix:
- terminate_group(): group-wide SIGTERM now, SIGKILL from a detached
backstop thread 5s later, used on every shutdown path (Drop and all
spawn-failure paths, which previously used bare start_kill and could
orphan the forked Python child too).
- Per-interface private TMPDIR under the runtime dir, wiped on every
(re)spawn — even a hard-killed or power-lost daemon can never
accumulate stale extraction dirs, and they land on the data disk
instead of the small tmpfs /tmp.
Verified: unit tests + the ignored mesh_service_connects_over_
reticulum_tcp_client integration test (spawns two real daemons over
loopback TCP through the changed spawn/terminate path), clean teardown
with no stray processes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Secret env used to merge into manifest.app.environment, landing in
'podman inspect' Config.Env on the API backend and — worse — as
plaintext Environment= lines in Quadlet unit files on disk. Now:
- expand_and_partition_env (container crate, pure + tested) expands
${KEY} placeholders and splits env into plain entries and
secret-bearing pairs. Plain entries that interpolate a secret
(btcpay's Password=${BTCPAY_DB_PASS} connection strings) are
tainted and travel as secrets too. Secret values themselves are
never expanded (a generated value containing '${' passes verbatim).
- values register as podman secrets: stdin (never argv/tempfile),
--replace, content-hash label to skip no-op rewrites; a per-app hash
cache in the orchestrator makes steady-state reconciles free of
podman secret calls. Registration goes through the runtime trait
(default no-op keeps mocks/docker inert).
- containers reference secrets by name: secret_env map in the libpod
create spec, Secret=<name>,type=env,target=<KEY> in Quadlet units.
Verified empirically on fleet podman 5.4.2: value absent from
inspect Config.Env, runtime injection works rootless.
- rotation detection: io.archipelago.secret-env-hash container label
(API) / the changed unit bytes (Quadlet). Pre-upgrade containers
lack the label, so every secret-bearing app recreates ONCE on the
first reconcile after deploy — deliberate, it scrubs the plaintext
secrets out of existing container configs. Data dirs untouched.
- docker dev fallback keeps plain -e injection (no secret store);
podman secrets persist across uninstall, matching the
preserve-credentials invariant (reinstall re-registers by hash).
In-container /proc/<pid>/environ is unchanged — env remains the
app-compat contract; the closed leaks are inspect output and unit
files on disk.
Tests: archipelago-container 61/61 (3 new: taint partition, verbatim
secrets, hash order-independence), archipelago container:: 160/160
(fedimint install test now asserts the secret arrives as a ref, not
env; quadlet render test asserts Secret=/Label= lines). NEEDS the
on-node gate re-run before the item counts as verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New container::image_verify gates PodmanClient::pull_image and the
dev-only DockerRuntime::pull_image. Signature claims classify three
ways: absent/empty (pull unverified, logged), the literal
'cosign://...' placeholder every fleet manifest carries today (same —
enforcement stays dormant until the signing ceremony ships real
values), or a declared signature, which must verify via
'cosign verify --key /etc/archipelago/cosign.pub
--insecure-ignore-tlog=true' (plus --allow-insecure-registry
--allow-http-registry for the HTTP mirror; flags checked against
cosign's own docs) before anything is fetched. Missing key, missing
cosign binary, timeout, or verification failure all hard-fail the
pull — a declared signature cannot be skipped on either runtime. Key
path overridable via ARCHIPELAGO_COSIGN_PUBKEY for tests/staging.
Deletes security::ImageVerifier: zero callers, blocking
std::process::Command on would-be async paths, and a fantasy
'cosign verify --signature' invocation (that flag belongs to
verify-blob).
Activation ships with the Workstream B ceremony, in order: pin
cosign.pub on nodes + install cosign, then publish real
image_signature values in the catalog.
Tests: archipelago-container 58/58 (5 new), archipelago container::
159/159, security check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mesh: verify_signature accepts a v2 preimage (t,v,ts,seq) alongside
legacy v1 (t,v,ts); signed_with_seq() is the v2 sender path, not yet
wired — senders stay v1 until the fleet verifies v2 (receivers
hard-drop bad sigs, so flipping send-side first would break
mixed-fleet alerts). Tests: v2 verify, v2 seq-tamper rejection,
v1 sign-then-set-seq compat.
- mesh listener: malformed radio-supplied DID shorter than the
'did🔑' prefix can no longer panic advert_name (slice -> .get()).
- auth: the pre-setup password123 dev login and the constant itself are
now #[cfg(debug_assertions)] — no release binary carries the bypass,
whatever its runtime config says.
- orchestrator: canned host-facts under #[cfg(test)] — awaiting real
subprocesses under tokio's paused test clock deadlocks against
auto-advanced timers (the old blocking detection only worked by never
yielding).
- drop two now-unused std::process::Command imports left by 4c75bb3d.
Tests: mesh 110/110 (incl. 2 new), api 68/68, container 159/159,
archipelago-container check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
verify_pending_update previously cleared the rollback marker on any
2xx/3xx from GET / — a release with a dead RPC API or broken podman
access passed and never rolled back. Verification now requires, in the
same attempt: the frontend via nginx, backend RPC liveness (an
unauthenticated POST /rpc/v1 — 401 proves the stack is up, 5xx/404/
refused fails it), and rootless podman reachability. A pre-loop check
also asserts the running binary's version matches what the marker says
was applied, catching a silent or half swap deterministically.
Per-app container assertions are deliberately excluded: the
pre-Quadlet service restart legitimately takes containers down and the
boot reconciler can need minutes for heavy apps — that would
false-rollback healthy updates. Revisit after the Phase-3 flip.
§B of the 1.8.0 hardening plan; update suite 38/38 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catalog- and manifest-supplied image refs reached pull_image without
ever passing the RPC boundary's validator — a malicious catalog entry
or manifest could pull from an arbitrary registry. The allowlist now
lives in container::image_policy (the RPC check delegates to it) and
both orchestrator pull sites (install_fresh and
ensure_resolved_source_available) refuse refs that fail it.
The shared policy accepts trusted-registry refs and registry-less
Docker Hub shorthand (grafana/grafana etc., used by 8 shipped
manifests — a registry-less ref cannot name an attacker host), and
rejects explicit non-allowlisted hosts, shell metacharacters, and
malformed refs. §A of the 1.8.0 hardening plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every production process spawn reachable from a tokio worker now uses
tokio::process: the install path's podman-port probe, the dependencies
disk check, factory-reset restart, config host-IP detection, the
orchestrator's host-facts helpers (resolve_dynamic_env and its call
sites made async to carry it through), and AutoRuntime's podman/docker
probes.
The FIPS transport probe is the special case: is_available() is a sync
trait method called from async route(), so instead of blocking ~50ms
on systemctl per stale-cache hit it now serves the cached value and
refreshes on a background thread (stale-while-revalidate) — bounded
staleness, zero stalled workers.
§C of the 1.8.0 hardening plan; container/transport/config/package
suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confirmed on real hardware: a genuine Heltec V4 RNode (firmware 1.86,
verified via rnodeconf --info against the same port) prints extra
boot/status chatter over the same serial line before answering KISS
commands. Replaying the exact probe bytes and timing budget our Rust
probe_rnode() uses showed DETECT_RESP arriving ~1.05s after the write —
past the old 800ms deadline, so a real, correctly-flashed, correctly-
responding RNode was being timed out and misclassified as "not an RNode".
2.5s leaves comfortable margin. Non-RNode devices (Meshcore/Meshtastic)
already budget ~5s each, so this doesn't meaningfully slow down detection
when the port turns out to be something else.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two MeshPeer rows were being created for one physical Reticulum node: a
radio twin (keyed by the RNS dest_hash, arch_pubkey_hex always None) and a
pseudo-federation twin (keyed by the archy ed25519 pubkey, created via the
generic identity-broadcast path meant for Meshcore/Meshtastic). The generic
path relies on bind_federation_twins matching both twins' advert_name, but
the Reticulum radio twin's display_name is deliberately never the identity
text — so the two rows could never merge, and the generic send path (keyed
off whichever twin the caller resolves) ended up looking up the archy
pubkey's prefix in ReticulumLink's `prefix_to_hash` map, which is only ever
populated with RNS dest_hash prefixes. Every send failed with "Unknown
Reticulum prefix ... peer hasn't announced yet", confirmed live between two
real nodes (archy-x250-exp / archy-x250-pa) that could see each other's
adverts but never exchange a message.
Unlike Meshcore/Meshtastic, Reticulum's ARCHY identity blob arrives in the
same announce event as the destination hash, so there's no ambiguity about
which peer it belongs to — bind it directly onto the RNS-hash-keyed radio
peer instead of relying on name-matching. Threaded through a new
`ParsedContact::arch_pubkey_hex` (None for Meshcore/Meshtastic, unchanged
behavior there) so `refresh_contacts` can set it on the correct peer row
without touching `bind_federation_twins`/`group_peer_twins`, which already
know how to collapse twins once they share an arch_pubkey_hex.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on real hardware (Heltec V4, RNode firmware): the board answers
the exact KISS detect probe correctly and instantly on a fresh port open,
but auto_detect_and_open (and open_preferred_path's unpinned branch) tried
Meshcore (~5s timeout) then Meshtastic (~5s timeout) first, leaving the
RNode firmware unresponsive by the time Reticulum's turn came ~10.6s later.
ReticulumLink::open() already gates its expensive daemon-spawn behind a
cheap ~1s probe_rnode check, so trying it first only costs ~1s extra when
the device turns out to be Meshcore/Meshtastic instead — the previous
comment's "most expensive, so goes last" reasoning only applies to a
successful match, not a failed one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Heltec V3/V4-class boards use the ESP32-S3's native USB-Serial-JTAG
peripheral (no discrete USB-UART bridge chip), which resets the chip on a
DTR/RTS transition — the same mechanism esptool uses to force bootloader
entry. The Meshcore/Meshtastic/RNode serial probes all open the port with
library defaults (DTR/RTS asserted), so each probe attempt was likely
rebooting a real, correctly RNode-flashed Heltec board mid-handshake,
surfacing as an endless "No supported mesh radio found" retry loop.
Deassert both lines immediately after open and let the board settle before
writing, in all three probe paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an additive, loopback-only TCP server/client interface to
reticulum-daemon and the Rust mesh wiring, alongside the unchanged serial
RNode path. Aurora (~/aurora) already speaks standard RNS/LXMF over plain
TCP by default, the same way Sideband already proved interop over LoRa
(docs/RETICULUM-TRANSPORT-PROGRESS.md gates #2/#3) — this closes the gap so
that interop is provable without scarce LoRa hardware, and gives archy a
path to eventually be dialed by an Aurora client.
Verified: daemon TCP transport round-trip, bidirectional LXMF DM against a
scripted RNS/LXMF stand-in for Aurora's Dart stack (content + dest-hash
match both directions), cargo check/test -p archipelago green (108 mesh
tests, 0 regressions), and a real MeshService::start() end-to-end test
spawning the daemon in TCP client mode with no serial probe.
TCP server mode is hard-gated to loopback in both Python and Rust — WAN/LAN
exposure is a deliberate future decision, not part of this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.
Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
check_for_updates now fetches the manifest as raw JSON and runs
trust::verify_detached before parsing: a tampered or wrong-signer
signature rejects the mirror outright, and unsigned manifests are
offered for MANUAL apply only — the 3 AM auto-apply scheduler refuses
them, closing the unattended remote-root hole (§A of the 1.8.0
hardening plan). UpdateState gains manifest_signed so the UI can
surface authenticity.
Publisher side: create-release.sh signs the manifest during the
release (ceremony, mnemonic via TTY/env only), publish-release-assets
hard-refuses to ship an unsigned manifest (grep + new 'ceremony
verify' cryptographic gate), and scripts/sign-manifest.sh covers
re-signing outside a release run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin RELEASE_ROOT_PUBKEY_HEX from the 2026-07-02 release-root signing ceremony
(signer did🔑z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur) so nodes verify
the publisher identity of the app-catalog. Sign releases/app-catalog.json in place.
Fix two floats that made the catalog unsignable: archy-btcpay-db manifest version
-> string, fedimint-clientd cpu_limit 0.25 -> 1 (u32). Add scripts/sign-catalog.sh
helper, the 1.8.0 release-hardening plan/tracker, and the commit-and-push project
rule in CLAUDE.md.
Backward-compatible: old binaries still accept the signed catalog; the pinned-anchor
binary ships in the next build/OTA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The boot reconciler only self-healed a fully-absent container for one of
8 hardcoded "required baseline" apps (bitcoin-knots, electrumx, lnd,
mempool*, filebrowser, fedimint-clientd) — every other genuinely-installed
app whose container went missing (crash, lost record, wedged teardown)
was left as Left("absent") forever, with no path back short of an
explicit manual reinstall.
Surfaced live: indeedhub's backend containers (minio/postgres/relay) went
absent on .116 and never recovered despite indeedhub still being
installed. By the time this code path runs, the app is already confirmed
NOT user-stopped and NOT user-uninstalled (both checked earlier in the
same function, backed by durable markers correctly cleared on
reinstall/start) — so gating self-heal further behind a hardcoded app-id
list was an unnecessary restriction, not a safety measure. An app the
user installed and never removed should come back on its own, same as
baseline services always have.
Deleted the now-dead is_required_baseline_app(); updated the test that
had locked in the old (wrong) behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pins RELEASE_ROOT_PUBKEY_HEX from the signing ceremony
(did🔑z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur). The
corresponding mnemonic is held offline by the publisher, never committed
or stored on any node/build host. Nodes built with this binary now verify
the app catalog's signature against this anchor instead of accepting any
signer; unsigned catalogs are still accepted during the migration window
per docs/workstream-b-signing-runbook.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reticulum daemon is a PyInstaller one-file binary: a bootloader parent
that forks the real Python process. `kill_on_drop`/`start_kill()` only SIGKILL
the bootloader, orphaning the forked child — which keeps holding the RNode
serial port. Across the listener's 30-min RX-stall reconnects this piled up
(observed 9 concurrent instances on a live node) all clutching /dev/ttyUSB0,
garbling the RNode so it stopped transmitting entirely.
Spawn the daemon as its own process-group leader (`process_group(0)`) and, on
drop, signal the whole group (SIGTERM for a clean RNode/socket release, then
SIGKILL as a hard backstop) so the forked child can never be orphaned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GoalDetail.vue, EasyHome.vue, and the backend's docker_packages.rs
metadata still pointed electrs-family app ids at the old electrs
icon (svg). Point them at electrumx.png like every other reference,
and delete the now-unused electrs.svg asset.
Peers that opt in via a new "Share Location" toggle in Settings
(server.set-location RPC) get plotted on other trusted peers' Mesh Map
with a distinct Archy-logo marker, separate from raw LoRa radio peers.
Location is persisted locally, carried in NodeStateSnapshot, and
propagated through federation sync/delta like other node state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- reticulum.rs: send_text_msg was lossy-UTF8-mangling binary CBOR control
envelopes (ReadReceipt etc.) before sending as LXMF text; base64-encode
with a marker instead, decoded losslessly on receive.
- typed_messages.rs: mesh.send-read-receipt fired automatically on every
chat view with no is_archy_peer gate, so viewing a message from a stock
(non-archy) LXMF peer auto-sent it an undecodable control envelope,
surfacing as garbage text right after whatever it just sent. Now a no-op
for non-archy peers.
- mesh/listener/mod.rs: RX_STALL_TIMEOUT was 300s and forced a full
auto-detect reconnect on any otherwise-healthy but quiet mesh link
(visible as "Connecting..." flapping); this also wiped Reticulum's
in-memory peer-address table every cycle, breaking messaging with peers
who hadn't re-announced in the window. Bumped to 1800s.
- reticulum.rs: persist the peer prefix/dest-hash/display-name table to
disk so a restart doesn't force every peer back to "Anonymous Peer"
until they re-announce.
- decode.rs/frames.rs: Meshcore was discarding the SNR its wire format
carries; wire it onto the peer record. Mesh.vue's signalBars() now falls
back to SNR-based bars when RSSI is unavailable (always true for
Meshcore); Reticulum has neither and correctly stays at 0/"no data".
- system/handlers.rs, dispatcher.rs: new system.get-hostname RPC + cert
regeneration (with a proper SAN) whenever server.set-name changes the
hostname, so HTTPS doesn't add a mismatch warning on top of the
self-signed one after a rename.
- AccountInfoSection.vue: surface the mDNS hostname + http/https links in
Settings (HTTPS needed for mic/camera secure-context features) — never
forced, both keep working.
- build-auto-installer-iso.sh: ship avahi-daemon so .local names actually
resolve on the LAN, and give the self-signed cert a real SAN instead of
a bare CN, both at image-build and install-time-fallback.
- Mesh.vue/MediaLightbox.vue/mesh-styles.css: mic/attach-stack no longer
closes on a plain hover-past; mesh images open in the shared lightbox
and have a real download button; lightbox close button moves to
bottom-center on mobile instead of under the status bar; mesh device
panel gets the same height/padding as its sibling tabs.
Verified: 108/108 mesh unit tests, deployed + confirmed healthy on
.116/.198/.228 (matching binary hash across all three), live Reticulum
messaging confirmed working end-to-end post-deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a TollGate row (Enabled/Disabled/Not installed) to the Home
dashboard's Network tile, polling the existing openwrt.get-status RPC
on the same cadence as the other network rows. Only rendered once an
OpenWrt router is actually configured, so nodes without one aren't
cluttered with an always-"Not configured" row.
Also fixes the underlying reason this could never have worked: nothing
in the OpenWrt Gateway flow ever persisted the router's host/credentials
server-side — the "connect" form only kept them in local component
state, so any no-args openwrt.get-status call (this new tile, and even
the Gateway page's own reload) always failed with "No router
configured" despite a fully working, provisioned router. Now
handle_openwrt_get_status saves the connection to router_config.json
whenever a host is explicitly passed in and the connection succeeds.
orchestrator_uninstall_app_ids("immich") only disabled the "immich" app_id
itself; "immich-postgres" and "immich-redis" (separate orchestrator-tracked
manifests, same pattern as mempool-api/archy-mempool-db) stayed enabled, so
the boot reconciler kept restarting their leftover stopped containers
forever after the generic uninstall path stopped them (.198, 2026-07-01 --
found while uninstalling immich to relieve disk I/O pressure competing with
a slow Bitcoin IBD).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- prod_orchestrator.rs: the boot reconciler's zombie-guard and start-failed
recreate paths (Created/Stopped/Exited states) had no attempt cap, unlike
health_monitor's independent restart tracker. A container whose entrypoint
fatally crashes right after `podman start` succeeds got stop+remove+
install_fresh'd every ~30s reconcile tick forever (portainer on .198,
2026-07-01: a DB schema newer than the pinned binary could read -- no
amount of recreating fixes that). Added a 5-attempts/30-minute circuit
breaker; once exhausted the container is left alone with an error! log
instead of looping, and an explicit install/start clears the counter.
- content_server.rs: serve_content now prunes a catalog entry whose backing
file is missing on disk, instead of leaving it advertised to every peer
forever with no way to distinguish "gone" from "transient failure."
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Several compounding bugs were blocking end-to-end TollGate provisioning
on OpenWrt 25.x (apk-native) routers:
- install_ipk's non-ar fallback assumed a flat tarball, but some .ipks are
a gzip tar of the three classic ipk members one level deep; it was
dumping debian-binary/data.tar.gz/control.tar.gz straight into / instead
of unpacking the real payload.
- Manually-extracted packages never ran their pending /etc/uci-defaults/*
scripts (that only happens through opkg/apk's own postinst bookkeeping),
so nothing ever created /etc/config/tollgate.
- uci_apply() never ensured the target config file existed first — `uci
set` fails outright on a config namespace nothing has created yet, which
is true for a package-defined one like "tollgate" (unlike wireless/
network/dhcp, which ship by default).
- The installed-check and restart_services looked for a binary/init script
named after the opkg package ("tollgate-module-basic-go"/"tollgate"),
but the real on-disk names are tollgate-wrt — so status always reported
"not installed" and service restarts silently no-op'd.
- provision_ssid used `uci add`, creating a new wifi-iface section (and
therefore a new duplicate broadcast SSID) on every provision call instead
of updating one in place.
Also adds a TollGateConfig.enabled field so the enable/disable state is
actually applied to the running service and the SSID's own broadcast
(stop + disable at boot, or start + enable), not just written to UCI.
On the frontend, the OpenWrt Gateway page's TollGate panel was read-only
once installed — add an edit form (price, step size, min steps, mint URL,
enabled toggle) that reuses the same idempotent provision-tollgate call.
Routers running MediaTek's proprietary mt_wifi SDK driver (e.g. GL.iNet)
never register with cfg80211/mac80211, so they have no `iw dev` entry and
no /sys/class/ieee80211 phy even though the radio is real and working —
find_wireless_iface was bailing with "No wireless radio found" on these.
Fall back to iwinfo's device listing, which abstracts over vendor backends
too, and to the vendor's iwpriv site-survey ioctl for scanning when iwinfo
itself can't trigger a scan on the interface.
- crash_recovery.rs: stack boot/runtime recovery (immich/indeedhub/netbird) now
requires the stack's core dependency container to exist before touching any
sibling, instead of firing on any leftover container. Fixes an infinite
120s-interval crash loop where orphan debris from a partial/failed install
(indeedhub-api with no indeedhub-postgres ever created) was repeatedly
force-restarted against a dependency that doesn't exist, which also blocked
a real reinstall via container name conflicts.
- AppSessionFrame.vue: the generic app-loading overlay and the ElectrumX
sync-in-progress overlay could render simultaneously (same z-index) during
launch. The sync screen is strictly more informative, so it now takes
precedence instead of the two stacking on top of each other.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- mempool-api now declares dependencies:[bitcoin:archival] directly, closing a
gap where installing it standalone (a legitimate direct orchestrator-install
target) bypassed the mempool umbrella's pruning gate entirely.
- New durable user-uninstalled marker (crash_recovery.rs, mirrors user_stopped)
fixes required-baseline-app self-heal (bitcoin-knots/electrumx/lnd/mempool/
etc.) resurrecting itself after an explicit uninstall survives a restart or
reboot, since the in-memory disabled set is wiped by every load_manifests().
- installed_version() (set_config.rs) no longer trusts a floating image tag
("latest") as the reported running version -- a stale local :latest cache
reported "latest" forever regardless of what latest had moved on to. Now
falls back to asking the Bitcoin backend directly via `bitcoind --version`
when the tag is floating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Master-plan backlog §10b/§10c: replace two per-app-hardcoded lookups with
generic, manifest-driven behavior so future apps are covered automatically
instead of needing a code edit.
- extract_lan_address (docker_packages.rs) now skips container-side ports
that are known non-HTTP (SSH, FTP, common DB ports) instead of blindly
taking podman's first-listed port. Fixes the whole class of bug the gitea
SSH-before-web static override was a one-off patch for.
- requires_unpruned_bitcoin (dependencies.rs) now checks the app's own
manifest for a `bitcoin:archival` dependency declaration first, falling
back to the old hardcoded id list. electrumx and mempool manifests now
declare it explicitly as the proof case.
869/869 Rust tests green, catalog drift clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three fixes:
1. Modem-preset authoritative: parse_config_lora_region now also decodes
modem_preset (field 2) alongside region, tracked as current_modem_preset.
ensure_lora_region's "region already set, don't touch it" branch (correct,
unchanged) now ALSO re-asserts LONG_FAST when a real observed preset has
drifted -- previously modem_preset only ever got written when region was
UNSET, so a radio with the right region but wrong preset was never fixed.
Only acts on an actually-observed wrong value (never speculative), so it
can't reboot-loop.
2. RX-stall watchdog: run_mesh_session now bails (triggering the existing
auto-reconnect path) if no frame has been successfully received in 5
minutes -- the existing consecutive_write_failures counter is blind to a
receive-only stall (writes can keep succeeding while inbound streaming is
wedged).
3. Hot-swap detection: spawn_mesh_listener now compares self_node_id across
session restarts and logs clearly when the physical radio itself changed
(not just an ordinary reconnect of the same board). Per-session device
state (contacts, current_region, etc.) was already naturally isolated
per-session (fresh struct each reconnect) -- nothing else needed clearing.
107/107 mesh tests pass (2 new: modem_preset decode + the
absent-field-defaults-to-LONG_FAST case).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New MeshDevicePanel.vue, added as a 4th/5th tab entry to activeTab/toolsTab/
mobileTab following the exact existing pattern (chat/bitcoin/deadman/
assistant/map). Shows firmware version, node ID, advert name, LoRa region,
channel, and device type -- firmware_version/self_node_id were already
server-side but never rendered; region is new (composed into MeshStatus from
MeshConfig.lora_region at read time, not part of the live session state).
Reboot button wired to the already-working mesh.reboot-radio RPC.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Backend: parse_mesh_packet now decodes MeshPacket.rx_snr (field 8, float) and
rx_rssi (field 12, int32), and a new POSITION_APP branch decodes Position.
latitude_i/longitude_i (fields 1/2, sfixed32) -- all field numbers confirmed
against the canonical meshtastic/protobufs mesh.proto, not guessed. Threaded
through ParsedContact -> refresh_contacts -> MeshPeer (mirroring how
pkc_capable was wired for #17), so mesh.peers now surfaces real rssi/snr/lat/
lon instead of always-null. Fixed a real bug found along the way:
update_node_info's unconditional contact replace would have silently wiped
any already-tracked signal/position data on the next NodeInfo packet -- now
preserves it.
Frontend: mesh.ts's updateNodePositionsFromPeers() feeds real position data
into the SAME nodePositions map MeshMap.vue already renders from (parallel to
the existing Coordinate/Alert-message path) -- MeshMap.vue itself needed zero
changes, it was already built for this.
105/105 mesh tests pass (4 new: rx_snr/rx_rssi decode, position decode +
incomplete-field handling, full packet_to_inbound_frame integration).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
