fedimint sat 'failed' for 7+ hours (exit 255 at 21:21, pre-gate) — the
reconciler repairs missing/drifted containers but never reset-fails +
starts a failed quadlet .service, and the health monitor can't see an
app whose container is gone. Concrete lever added to the Tier 2
container-flapping item.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
capabilities:[] was latent — the long-lived legacy container predated
strict manifest enforcement, so nothing noticed that a recreate against
this manifest produces a root process without DAC_OVERRIDE that
EACCESes on upload/encoded-video and crash-loops (49 systemd restarts
on .228 when the 2026-07-05 secret-env migration finally recreated
it). Any reinstall or reboot-repair would have tripped the same wire.
Cap set mirrors immich-postgres minus SETUID/SETGID.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Secret env used to merge into manifest.app.environment, landing in
'podman inspect' Config.Env on the API backend and — worse — as
plaintext Environment= lines in Quadlet unit files on disk. Now:
- expand_and_partition_env (container crate, pure + tested) expands
${KEY} placeholders and splits env into plain entries and
secret-bearing pairs. Plain entries that interpolate a secret
(btcpay's Password=${BTCPAY_DB_PASS} connection strings) are
tainted and travel as secrets too. Secret values themselves are
never expanded (a generated value containing '${' passes verbatim).
- values register as podman secrets: stdin (never argv/tempfile),
--replace, content-hash label to skip no-op rewrites; a per-app hash
cache in the orchestrator makes steady-state reconciles free of
podman secret calls. Registration goes through the runtime trait
(default no-op keeps mocks/docker inert).
- containers reference secrets by name: secret_env map in the libpod
create spec, Secret=<name>,type=env,target=<KEY> in Quadlet units.
Verified empirically on fleet podman 5.4.2: value absent from
inspect Config.Env, runtime injection works rootless.
- rotation detection: io.archipelago.secret-env-hash container label
(API) / the changed unit bytes (Quadlet). Pre-upgrade containers
lack the label, so every secret-bearing app recreates ONCE on the
first reconcile after deploy — deliberate, it scrubs the plaintext
secrets out of existing container configs. Data dirs untouched.
- docker dev fallback keeps plain -e injection (no secret store);
podman secrets persist across uninstall, matching the
preserve-credentials invariant (reinstall re-registers by hash).
In-container /proc/<pid>/environ is unchanged — env remains the
app-compat contract; the closed leaks are inspect output and unit
files on disk.
Tests: archipelago-container 61/61 (3 new: taint partition, verbatim
secrets, hash order-independence), archipelago container:: 160/160
(fedimint install test now asserts the secret arrives as a ref, not
env; quadlet render test asserts Secret=/Label= lines). NEEDS the
on-node gate re-run before the item counts as verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New container::image_verify gates PodmanClient::pull_image and the
dev-only DockerRuntime::pull_image. Signature claims classify three
ways: absent/empty (pull unverified, logged), the literal
'cosign://...' placeholder every fleet manifest carries today (same —
enforcement stays dormant until the signing ceremony ships real
values), or a declared signature, which must verify via
'cosign verify --key /etc/archipelago/cosign.pub
--insecure-ignore-tlog=true' (plus --allow-insecure-registry
--allow-http-registry for the HTTP mirror; flags checked against
cosign's own docs) before anything is fetched. Missing key, missing
cosign binary, timeout, or verification failure all hard-fail the
pull — a declared signature cannot be skipped on either runtime. Key
path overridable via ARCHIPELAGO_COSIGN_PUBKEY for tests/staging.
Deletes security::ImageVerifier: zero callers, blocking
std::process::Command on would-be async paths, and a fantasy
'cosign verify --signature' invocation (that flag belongs to
verify-blob).
Activation ships with the Workstream B ceremony, in order: pin
cosign.pub on nodes + install cosign, then publish real
image_signature values in the catalog.
Tests: archipelago-container 58/58 (5 new), archipelago container::
159/159, security check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mesh: verify_signature accepts a v2 preimage (t,v,ts,seq) alongside
legacy v1 (t,v,ts); signed_with_seq() is the v2 sender path, not yet
wired — senders stay v1 until the fleet verifies v2 (receivers
hard-drop bad sigs, so flipping send-side first would break
mixed-fleet alerts). Tests: v2 verify, v2 seq-tamper rejection,
v1 sign-then-set-seq compat.
- mesh listener: malformed radio-supplied DID shorter than the
'did🔑' prefix can no longer panic advert_name (slice -> .get()).
- auth: the pre-setup password123 dev login and the constant itself are
now #[cfg(debug_assertions)] — no release binary carries the bypass,
whatever its runtime config says.
- orchestrator: canned host-facts under #[cfg(test)] — awaiting real
subprocesses under tokio's paused test clock deadlocks against
auto-advanced timers (the old blocking detection only worked by never
yielding).
- drop two now-unused std::process::Command imports left by 4c75bb3d.
Tests: mesh 110/110 (incl. 2 new), api 68/68, container 159/159,
archipelago-container check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Was implicit across the Phase-3 Quadlet flip and Workstream F; now one
consolidated pre-tag item with the lever list, an observability proposal
(per-app restart counter + flap log line), and an already-landed list so
nothing gets re-done.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
verify_pending_update previously cleared the rollback marker on any
2xx/3xx from GET / — a release with a dead RPC API or broken podman
access passed and never rolled back. Verification now requires, in the
same attempt: the frontend via nginx, backend RPC liveness (an
unauthenticated POST /rpc/v1 — 401 proves the stack is up, 5xx/404/
refused fails it), and rootless podman reachability. A pre-loop check
also asserts the running binary's version matches what the marker says
was applied, catching a silent or half swap deterministically.
Per-app container assertions are deliberately excluded: the
pre-Quadlet service restart legitimately takes containers down and the
boot reconciler can need minutes for heavy apps — that would
false-rollback healthy updates. Revisit after the Phase-3 flip.
§B of the 1.8.0 hardening plan; update suite 38/38 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confirmed on the just-packaged binary: `archy-rnodeconf --info` printed
everything correctly, then crashed with NameError: name 'exit' is not
defined and returned exit code 1. rnodeconf.py's own graceful_exit() calls
the bare exit() builtin, which is only ever defined by site.py for
interactive Python — a frozen PyInstaller app skips that init, so any
bundled script relying on it hits this the moment it tries to quit cleanly,
after the real work already succeeded. Classic, well-documented PyInstaller
gotcha; the standard fix is a runtime hook pre-defining exit/quit as
sys.exit before the bundled script's own code runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RNS's own rnodeconf utility (frequency/bandwidth/spreading-factor/coding-rate
read+write, firmware signature verification, board bootstrap) has been the
one tool that reliably diagnoses real RNode hardware — it's what finally
proved two live nodes were silently running at different spreading factors
(SF5 vs SF10, invisible from any of our own probe/logging code, and the
actual reason two correctly-flashed radios could detect each other's RF but
never decode a packet). Every node should have it, not just whichever one an
agent happened to build a throwaway venv on to debug a specific incident.
- reticulum-daemon/build.sh: also PyInstaller-package archy-rnodeconf
alongside the existing archy-reticulum-daemon, same --collect-submodules/
-d noarchive flags (same RNS.Interfaces __all__-glob requirement applies).
- scripts/deploy-to-target.sh: actually wire both packaged binaries into the
live deploy path (neither was wired in before — a pre-existing gap noted
in docs/RETICULUM-TRANSPORT-PROGRESS.md; this is why reticulum-daemon
previously only worked via manual dev-venv setup on rsync-deployed nodes,
not the ISO-imaged ones). Non-fatal on build/deploy failure — archipelago
already falls back to its dev-venv path if the packaged binary is absent.
Also installs the missing `python3.<minor>-venv` package when needed
(same "ensurepip not available" gap hit manually twice this session).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Catalog- and manifest-supplied image refs reached pull_image without
ever passing the RPC boundary's validator — a malicious catalog entry
or manifest could pull from an arbitrary registry. The allowlist now
lives in container::image_policy (the RPC check delegates to it) and
both orchestrator pull sites (install_fresh and
ensure_resolved_source_available) refuse refs that fail it.
The shared policy accepts trusted-registry refs and registry-less
Docker Hub shorthand (grafana/grafana etc., used by 8 shipped
manifests — a registry-less ref cannot name an attacker host), and
rejects explicit non-allowlisted hosts, shell metacharacters, and
malformed refs. §A of the 1.8.0 hardening plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Server.vue rendered the backend-generated WireGuard peer QR with raw
v-html while the analogous TOTP QR was DOMPurify-sanitized — both now
use the same svg-profile sanitizer. Mesh.vue's 5s poll interval gets
the same start-guard as the arch poll (no leak on double-mount) and is
nulled on unmount. curatedApps.ts catalog fetches no longer fail
silently: each failed source logs a console.warn, including the final
all-sources-failed fallback to the hardcoded list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every production process spawn reachable from a tokio worker now uses
tokio::process: the install path's podman-port probe, the dependencies
disk check, factory-reset restart, config host-IP detection, the
orchestrator's host-facts helpers (resolve_dynamic_env and its call
sites made async to carry it through), and AutoRuntime's podman/docker
probes.
The FIPS transport probe is the special case: is_available() is a sync
trait method called from async route(), so instead of blocking ~50ms
on systemctl per stale-cache hit it now serves the cached value and
refreshes on a background thread (stale-while-revalidate) — bounded
staleness, zero stalled workers.
§C of the 1.8.0 hardening plan; container/transport/config/package
suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confirmed on real hardware: a genuine Heltec V4 RNode (firmware 1.86,
verified via rnodeconf --info against the same port) prints extra
boot/status chatter over the same serial line before answering KISS
commands. Replaying the exact probe bytes and timing budget our Rust
probe_rnode() uses showed DETECT_RESP arriving ~1.05s after the write —
past the old 800ms deadline, so a real, correctly-flashed, correctly-
responding RNode was being timed out and misclassified as "not an RNode".
2.5s leaves comfortable margin. Non-RNode devices (Meshcore/Meshtastic)
already budget ~5s each, so this doesn't meaningfully slow down detection
when the port turns out to be something else.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two MeshPeer rows were being created for one physical Reticulum node: a
radio twin (keyed by the RNS dest_hash, arch_pubkey_hex always None) and a
pseudo-federation twin (keyed by the archy ed25519 pubkey, created via the
generic identity-broadcast path meant for Meshcore/Meshtastic). The generic
path relies on bind_federation_twins matching both twins' advert_name, but
the Reticulum radio twin's display_name is deliberately never the identity
text — so the two rows could never merge, and the generic send path (keyed
off whichever twin the caller resolves) ended up looking up the archy
pubkey's prefix in ReticulumLink's `prefix_to_hash` map, which is only ever
populated with RNS dest_hash prefixes. Every send failed with "Unknown
Reticulum prefix ... peer hasn't announced yet", confirmed live between two
real nodes (archy-x250-exp / archy-x250-pa) that could see each other's
adverts but never exchange a message.
Unlike Meshcore/Meshtastic, Reticulum's ARCHY identity blob arrives in the
same announce event as the destination hash, so there's no ambiguity about
which peer it belongs to — bind it directly onto the RNS-hash-keyed radio
peer instead of relying on name-matching. Threaded through a new
`ParsedContact::arch_pubkey_hex` (None for Meshcore/Meshtastic, unchanged
behavior there) so `refresh_contacts` can set it on the correct peer row
without touching `bind_federation_twins`/`group_peer_twins`, which already
know how to collapse twins once they share an arch_pubkey_hex.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on real hardware (Heltec V4, RNode firmware): the board answers
the exact KISS detect probe correctly and instantly on a fresh port open,
but auto_detect_and_open (and open_preferred_path's unpinned branch) tried
Meshcore (~5s timeout) then Meshtastic (~5s timeout) first, leaving the
RNode firmware unresponsive by the time Reticulum's turn came ~10.6s later.
ReticulumLink::open() already gates its expensive daemon-spawn behind a
cheap ~1s probe_rnode check, so trying it first only costs ~1s extra when
the device turns out to be Meshcore/Meshtastic instead — the previous
comment's "most expensive, so goes last" reasoning only applies to a
successful match, not a failed one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Heltec V3/V4-class boards use the ESP32-S3's native USB-Serial-JTAG
peripheral (no discrete USB-UART bridge chip), which resets the chip on a
DTR/RTS transition — the same mechanism esptool uses to force bootloader
entry. The Meshcore/Meshtastic/RNode serial probes all open the port with
library defaults (DTR/RTS asserted), so each probe attempt was likely
rebooting a real, correctly RNode-flashed Heltec board mid-handshake,
surfacing as an endless "No supported mesh radio found" retry loop.
Deassert both lines immediately after open and let the board settle before
writing, in all three probe paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an additive, loopback-only TCP server/client interface to
reticulum-daemon and the Rust mesh wiring, alongside the unchanged serial
RNode path. Aurora (~/aurora) already speaks standard RNS/LXMF over plain
TCP by default, the same way Sideband already proved interop over LoRa
(docs/RETICULUM-TRANSPORT-PROGRESS.md gates #2/#3) — this closes the gap so
that interop is provable without scarce LoRa hardware, and gives archy a
path to eventually be dialed by an Aurora client.
Verified: daemon TCP transport round-trip, bidirectional LXMF DM against a
scripted RNS/LXMF stand-in for Aurora's Dart stack (content + dest-hash
match both directions), cargo check/test -p archipelago green (108 mesh
tests, 0 regressions), and a real MeshService::start() end-to-end test
spawning the daemon in TCP client mode with no serial probe.
TCP server mode is hard-gated to loopback in both Python and Rust — WAN/LAN
exposure is a deliberate future decision, not part of this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.
Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nostr bridge derived the caller from the launcher's own URL and
never checked event.origin, so any co-resident iframe could pull the
node's nostr pubkey or use nip04/nip44 decrypt as an oracle while an
app was open. The bridge now rejects senders whose real origin doesn't
match the open app's origin, and every identity-sensitive method
(getPublicKey, signEvent, encrypt/decrypt) requires user consent or a
remembered per-origin approval — previously only signEvent did.
share-to-mesh in App.vue likewise accepted messages from any sender
and force-navigated to /mesh with an attacker-staged CID; it now
requires same-origin, matching Chat.vue's existing handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
check_for_updates now fetches the manifest as raw JSON and runs
trust::verify_detached before parsing: a tampered or wrong-signer
signature rejects the mirror outright, and unsigned manifests are
offered for MANUAL apply only — the 3 AM auto-apply scheduler refuses
them, closing the unattended remote-root hole (§A of the 1.8.0
hardening plan). UpdateState gains manifest_signed so the UI can
surface authenticity.
Publisher side: create-release.sh signs the manifest during the
release (ceremony, mnemonic via TTY/env only), publish-release-assets
hard-refuses to ship an unsigned manifest (grep + new 'ceremony
verify' cryptographic gate), and scripts/sign-manifest.sh covers
re-signing outside a release run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin RELEASE_ROOT_PUBKEY_HEX from the 2026-07-02 release-root signing ceremony
(signer did🔑z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur) so nodes verify
the publisher identity of the app-catalog. Sign releases/app-catalog.json in place.
Fix two floats that made the catalog unsignable: archy-btcpay-db manifest version
-> string, fedimint-clientd cpu_limit 0.25 -> 1 (u32). Add scripts/sign-catalog.sh
helper, the 1.8.0 release-hardening plan/tracker, and the commit-and-push project
rule in CLAUDE.md.
Backward-compatible: old binaries still accept the signed catalog; the pinned-anchor
binary ships in the next build/OTA.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BackButton is presentational-only (emits click, parent wires navigation)
per its own doc comment, but OpenWrtGateway.vue rendered it with no
@click handler at all -- clicking it did nothing. Added useRouter +
goBack() (-> the 'server' route, matching the page's location under
views/server/), same pattern as PeerFiles.vue/CloudFolder.vue.
Router-detection (openwrt.scan) spot-checked live: RPC plumbing works
end-to-end and returns a valid response, but no physical OpenWrt device
was on hand to confirm a true-positive detection. Also noted:
detect::scan_subnet does blocking TCP/SSH calls inside an async fn with
no .await points -- not proven to cause a real issue yet, but worth
hardening (spawn_blocking or async I/O) before a large subnet scan is
exercised for real.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Web5ConnectedNodes.vue declared nodesContainerRef but never consumed it
(the controller-nav system scans [data-controller-container] globally,
no other view uses a per-component ref for it) — broke the vue-tsc build.
MeshMap.test.ts's mocked mesh store predated federatedPositions (added
earlier this session for the Mesh Map federated-node feature) and crashed
on mount. Found live merging PR#67 (reticulum) + UI/UX work +
archy-openwrt into main for a combined fleet deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Mesh: fix 920-1280px bottom margin (phantom mobile-nav reservation
leaking into the desktop-sidebar range), let the mesh view scale to
full width on wide screens instead of capping at 1600px, and make the
Device panel collapsible on desktop (previously mobile-only)
- Search/controller-nav: a global gamepad/keyboard-nav feature was
auto-clicking "the next button in the DOM" on Enter in any text input,
which cleared the mesh peer search and popped the sideload modal from
the App Store/My Apps search boxes. Opt out via data-controller-no-submit
on all filter inputs; bump the mesh clear button's touch target
- Modals: several (sideload, credential, Lightning channel open, identity
create) used ad-hoc blue buttons and non-fullscreen backdrops that only
covered the main content area, not the sidebar. Teleport them to body,
unify backdrop/button theming to the dark+orange convention, fix the
sideload modal's square bottom corners on desktop, and standardize
close buttons to the ghost-icon style
- Web5: remove the redundant/dead "Messages" tab from Connected Nodes
(its deep-link was unreachable dead code); fix the "view message" toast
to actually open the Archipelago channel instead of silently failing to
match a LoRa peer; make identity rows responsive via a container query
(viewport-based breakpoints don't work in the page's 2-column grid) and
right-justify their action icons; collapse DID/DHT/Wallet/Nostr/Connected
Nodes by default on mobile
- Apps/App Store: match the search bar and sideload button's height,
padding, and background to the mode-switcher tabs beside them
- Mesh chat: keep the compose input focused after sending
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The boot reconciler only self-healed a fully-absent container for one of
8 hardcoded "required baseline" apps (bitcoin-knots, electrumx, lnd,
mempool*, filebrowser, fedimint-clientd) — every other genuinely-installed
app whose container went missing (crash, lost record, wedged teardown)
was left as Left("absent") forever, with no path back short of an
explicit manual reinstall.
Surfaced live: indeedhub's backend containers (minio/postgres/relay) went
absent on .116 and never recovered despite indeedhub still being
installed. By the time this code path runs, the app is already confirmed
NOT user-stopped and NOT user-uninstalled (both checked earlier in the
same function, backed by durable markers correctly cleared on
reinstall/start) — so gating self-heal further behind a hardcoded app-id
list was an unnecessary restriction, not a safety measure. An app the
user installed and never removed should come back on its own, same as
baseline services always have.
Deleted the now-dead is_required_baseline_app(); updated the test that
had locked in the old (wrong) behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.5's 5x gate done: 5/5 iterations, all technically FAIL per run-gate.sh's
tally but only from .5's permanent pruned-bitcoin ceiling (accepted going
in); down to 2 failures/iteration by the end. Found + fixed a real hang
(lnd cached a dead bitcoin-knots IP after a restart) live mid-run.
Separately found a real boot-reconciler bug via indeedhub going stuck on
.116: any genuinely-installed-but-fully-absent app was left stuck forever
unless it was one of 8 hardcoded "baseline" apps. Fix tracked, code change
in the shared working tree pending test confirmation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pins RELEASE_ROOT_PUBKEY_HEX from the signing ceremony
(did🔑z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur). The
corresponding mnemonic is held offline by the publisher, never committed
or stored on any node/build host. Nodes built with this binary now verify
the app catalog's signature against this anchor instead of accepting any
signer; unsigned catalogs are still accepted during the migration window
per docs/workstream-b-signing-runbook.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reticulum daemon is a PyInstaller one-file binary: a bootloader parent
that forks the real Python process. `kill_on_drop`/`start_kill()` only SIGKILL
the bootloader, orphaning the forked child — which keeps holding the RNode
serial port. Across the listener's 30-min RX-stall reconnects this piled up
(observed 9 concurrent instances on a live node) all clutching /dev/ttyUSB0,
garbling the RNode so it stopped transmitting entirely.
Spawn the daemon as its own process-group leader (`process_group(0)`) and, on
drop, signal the whole group (SIGTERM for a clean RNode/socket release, then
SIGKILL as a hard backstop) so the forked child can never be orphaned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The daemon ships as a PyInstaller one-file binary; its direct parent is the
bootloader, which the Rust supervisor (mesh/reticulum.rs Drop) stops via
start_kill() == SIGKILL. SIGKILL can't be forwarded, so the Python child was
orphaned on every link recreation and kept holding the RNode serial port.
These stale daemons piled up (9 seen on one node), all clutching /dev/ttyUSB0
and garbling the RNode so it silently stopped transmitting (txb frozen,
interface status False).
Set PR_SET_PDEATHSIG(SIGTERM) at daemon startup so the kernel signals us when
the parent exits; our existing SIGTERM handler then shuts down cleanly and
frees the port. Linux-only, best-effort, no-op elsewhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
While the .5 gate ran: confirmed no legacy multi-container stacks remain
(workstream A tail fully closed), reframed the "30 apps zero coverage"
claim as stale (all apps get generic baseline coverage via
all-apps-lifecycle/matrix, real gap is 34 apps lacking app-specific
assertions), and discovered tests/multinode/smoke.sh already exists and
ran it live against .116<->.228: federation pairing/FIPS/content-browse
all confirmed working, but found + root-caused a real tombstone bug
(federation.remove-node silently swallows tombstone-write failures,
letting removed peers get re-added by background sync). Not fixed yet —
federation/trust code, needs a careful fix, not a blind one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The prior fix's loop `container_installed "$c" && echo "$c"` makes the
function's own exit status the exit status of its LAST array entry. If
that entry isn't installed on this node (e.g. required-stack-destructive's
array ends with mempool-api, absent on .5), the whole function reports
failure even though earlier entries matched fine — and under bats' set -e,
`targets="$(installed_required_containers)"` then aborts the test outright.
required-stack.bats got lucky (its array happens to end with an installed
container) but has the identical latent bug. Caught live on .5's iteration
3 of the multinode-pass gate run. Add explicit `return 0`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GoalDetail.vue, EasyHome.vue, and the backend's docker_packages.rs
metadata still pointed electrs-family app ids at the old electrs
icon (svg). Point them at electrumx.png like every other reference,
and delete the now-unused electrs.svg asset.
Same class of bug as required-stack.bats: hardcoded required_containers
included mempool/mempool-api unconditionally, so a node without the
mempool stack (e.g. .5) hard-fails restarting a container that was never
installed, and waits out full 180-240s timeouts probing endpoints that
will never come up. Likely explains .5's abnormally long (2216s) iteration
1 runtime during the current multinode-pass run. Same skip-if-absent fix
as the prior commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live during the .5 multinode-pass run: this suite was hardcoded to
.116's exact app bundle (including the mempool stack), so any node missing
an app hard-failed instead of skipping — and a missing local fail() helper
(present in 3 sibling bats files, absent here) masked the real error as
"command not found" (exit 127). Add the same skip-if-absent idiom already
used in mempool.bats per-app, and define fail() locally like the others.
Verified: skips cleanly on .116 (no bitcoin-knots here), still exercises
real checks for apps that are installed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.198 IBD/pruned blocker → user chose swap over wait/hardware. .116 ruled
out (no bitcoin container), .120 ruled out (reserved for another dev). .5
(archy-x250-beta) is fully synced despite also being sub-1TB/pruned;
bootstrapped bats+jq and launched the 5x destructive gate there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reset-failed 2 stale dead-unit records on .198, confirmed nginx lnd proxy
target is correct. Hit a genuine blocker needing a user decision: .198's
448GB disk is below the 1TB archival threshold so it runs pruned bitcoin,
currently only 21% through IBD — the multinode plan's precondition requires
pruned:false + fully synced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
