load_manifests() only ran at startup, so a manifest published via the
signed catalog sat dormant until the next service restart (found while
shipping the strfry manifest fixes: package.check-updates refreshed the
cache but the orchestrator kept rendering the old overlay).
- refresh_catalog now reports whether the cached bytes actually changed
(write_cache skips identical rewrites), so unchanged hourly polls
don't churn the manifest map.
- New ContainerOrchestrator::reload_manifests (default no-op); prod
delegates to load_manifests — the rebuild is atomic under the state
write lock, and the reconciler's durable user-stopped/uninstalled
marker filters make a mid-session reload equivalent to the restart
path that already runs on every boot.
- package.check-updates reloads on change and reports catalog_changed +
manifests_reloaded; the hourly update-scheduler tick (and its startup
refresh) do the same.
Tests: app_catalog 5/5 (new write_cache changed-detection test),
reconcile 16/16, knows_app 1/1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.install routed to the manifest-driven orchestrator only for a
hardcoded per-app allowlist; every other app fell through to the legacy
flow, which ignores manifests entirely and creates a bare container with
no ports or volumes (strfry crash-looped this way on .228, 2026-07-09).
New ContainerOrchestrator::knows_app(app_id) (default false; prod checks
its loaded-manifest map, disk + signed-catalog overlay). The install gate
is now allowlist OR knows_app — no per-app Rust for manifest-driven apps,
matching the packaging invariant. Apps without a manifest keep the legacy
flow including the container-exists adopt block; the unknown-app_id →
legacy fallback inside the orchestrator branch is unchanged.
Tests: knows_app_reflects_loaded_manifests + install suite 37/37.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reconciler doesn't take the RPC layer's per-app FIFO op lock (known
limit of 891cbba4): between a restart worker's stop and start halves it saw
the mempool frontend "missing", repair-recreated it behind systemd's back,
killed the worker's fresh container 11s after start, and left the unit down
for ~3.5 min until the next heal — gate test 123 measured exactly that
window (.228 iteration 3, 2026-07-09).
New crate::app_ops module owns the op-lock registry + stack member table
(runtime.rs and dependencies.rs now delegate) so the reconciler can probe
lifecycle_op_in_flight(app_id) — covering both the app's own key and its
owning stack package — and skip that app for the cycle. The ownership-sweep
podman restart gets the same guard. Health monitor is a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lnd resolves the bitcoin RPC address once at startup and never re-resolves;
when bitcoin-core/knots restarts (new container IP) lnd spins on "dial tcp
<old-ip>:8332: no route to host" and its own RPC wedges until lnd restarts
(gate lnd getinfo test 75 failed every iteration on .228, 2026-07-09;
hardening plan §C cascade item). electrumx is already restarted by the
version-switch path — lnd was the only dependent left stranded.
After a successful package.start/restart of a bitcoin backend, restart
running, not-user-stopped address-caching dependents under their own app-op
lock (lock order backend→dependent only, so no deadlock).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.stop/restart on orchestrator-managed stacks (immich, indeedhub,
btcpay, netbird, mempool) addressed live CONTAINER names; unknown app ids
fell through to raw podman stop/start, racing systemd's --rm cleanup of the
quadlet unit — the container vanished on stop and the start half found
nothing (immich RESTART FAIL + indeedhub stuck restarting, gate 2026-07-09).
Now stack ops resolve member APP ids (reverse start order for stop) so the
orchestrator drives the quadlet .service; user-stop markers are written for
member app ids too (covers absent containers), and the orchestrator stop
fan-out treats an already-missing container as stopped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/var/log/archipelago/container-installs.log has been 0 bytes since April:
the service sandbox (ProtectSystem=strict class) leaves /var/log
read-only for the unit, the open() fails, and install_log() drops every
line by design (fire-and-forget). These START/STOP/RESTART/FAIL
breadcrumbs are the primary forensic trail for gate failures, so mirror
every line to tracing (journald) unconditionally and keep the file
append as best-effort for hosts where the path is writable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.start/stop/restart reply immediately and run their multi-container
sequences in spawned tasks with no coordination. Back-to-back RPCs on the
same app interleave those sequences: gate run D (.228, 2026-07-09 04:28)
fired stop→start→restart on mempool; the start's member bring-up raced the
still-running stop's member shutdown — archy-mempool-db was stopped 2s
after it started and the stack finished with ZERO containers (tests 95/123/
124). A user double-clicking restart in the UI can do the same.
Workers now take a per-app tokio Mutex (fair/FIFO, keyed by the normalized
orchestrator app id) as their first await, so queued ops run in RPC arrival
order and the final state matches the last request. The RPC reply stays
immediate.
Known limit: container lists are still computed at RPC time (pre-lock), so
a stop landing mid-start can still return 'No containers found' — it can no
longer interleave/corrupt, only error. Reconciler/health-monitor restarts
don't take the lock yet (separate paths, same candidate follow-up).
Also from run D, filed separately: jellyfin/vaultwarden (legacy containers
on a quadlet node) stop fine but the package state sticks at 'stopping' —
'Failed to stop jellyfin.service: Unit not loaded' error path suspected.
Tests: runtime 9/9; archipelago builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second instance of the stop→start destruction class from tonight's gate
runs (.228): package.stop of a multi-container stack marks every member
user-stopped and quadlet-removes their containers; package.start's fallback
then resurrected only the bare package id, leaving the other members absent
AND user-stopped — indeedhub lost all 7 containers this way (mempool needed
its umbrella alias for the same reason).
- New stack_member_app_ids map (real manifest app ids, dependency order) —
used by the start fallback when a stack has no live containers, and by
package.restart instead of hard-failing "No containers found".
- orchestrator_uninstall_app_ids grows the missing indeedhub arm (same
reconciler-ghost rationale as the immich arm).
- do_orchestrator_package_start no longer aborts the whole sequence when one
member's unknown-app-id fallback fails (the #73/#74 wish) — collects and
continues.
NOTE: extends the known §G per-app-map debt (tracker updated) — the proper
home for stack membership is a manifest field; parked post-tag by design.
Tests: package:: 48/48 incl. new fallback + indeedhub-uninstall coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the .228 lifecycle gate (2026-07-08, first quadlet-mode run):
stop→start on the legacy `mempool` umbrella app id destroyed the mempool
deployment. Under quadlet, package.stop removes the containers; package.start
then failed with `unknown app_id: mempool` because load_manifests drops the
umbrella manifest whenever the three split-stack members are loaded — leaving
nothing to recreate from and the stack down.
Two fixes:
- prod_orchestrator start/stop/restart now resolve the historical
`mempool`/`mempool-web` id to the split members (archy-mempool-db,
mempool-api, archy-mempool-web) whenever the umbrella manifest was dropped —
the same alias install already implements ("installing mempool assembles the
split stack"). Restart falls back to start for members whose container/unit
is gone. Legacy umbrella-only nodes are unaffected (alias inactive while the
umbrella manifest is live).
- is_missing_container_error (3 sites) now recognizes podman 5.x's
`no such object` inspect phrasing, so a genuinely absent container is
classified as missing instead of surfacing as a hard inspect error.
Tests: 2 new alias tests + 2 classifier tests; prod_orchestrator suite 61/61
green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mesh: verify_signature accepts a v2 preimage (t,v,ts,seq) alongside
legacy v1 (t,v,ts); signed_with_seq() is the v2 sender path, not yet
wired — senders stay v1 until the fleet verifies v2 (receivers
hard-drop bad sigs, so flipping send-side first would break
mixed-fleet alerts). Tests: v2 verify, v2 seq-tamper rejection,
v1 sign-then-set-seq compat.
- mesh listener: malformed radio-supplied DID shorter than the
'did🔑' prefix can no longer panic advert_name (slice -> .get()).
- auth: the pre-setup password123 dev login and the constant itself are
now #[cfg(debug_assertions)] — no release binary carries the bypass,
whatever its runtime config says.
- orchestrator: canned host-facts under #[cfg(test)] — awaiting real
subprocesses under tokio's paused test clock deadlocks against
auto-advanced timers (the old blocking detection only worked by never
yielding).
- drop two now-unused std::process::Command imports left by 4c75bb3d.
Tests: mesh 110/110 (incl. 2 new), api 68/68, container 159/159,
archipelago-container check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catalog- and manifest-supplied image refs reached pull_image without
ever passing the RPC boundary's validator — a malicious catalog entry
or manifest could pull from an arbitrary registry. The allowlist now
lives in container::image_policy (the RPC check delegates to it) and
both orchestrator pull sites (install_fresh and
ensure_resolved_source_available) refuse refs that fail it.
The shared policy accepts trusted-registry refs and registry-less
Docker Hub shorthand (grafana/grafana etc., used by 8 shipped
manifests — a registry-less ref cannot name an attacker host), and
rejects explicit non-allowlisted hosts, shell metacharacters, and
malformed refs. §A of the 1.8.0 hardening plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every production process spawn reachable from a tokio worker now uses
tokio::process: the install path's podman-port probe, the dependencies
disk check, factory-reset restart, config host-IP detection, the
orchestrator's host-facts helpers (resolve_dynamic_env and its call
sites made async to carry it through), and AutoRuntime's podman/docker
probes.
The FIPS transport probe is the special case: is_available() is a sync
trait method called from async route(), so instead of blocking ~50ms
on systemctl per stale-cache hit it now serves the cached value and
refreshes on a background thread (stale-while-revalidate) — bounded
staleness, zero stalled workers.
§C of the 1.8.0 hardening plan; container/transport/config/package
suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
§C of the 1.8.0 hardening plan: persistence writes whose Results were
silently dropped now log a warn/error with context (mesh contact
blocklist, scheduler state, content catalog, container registry,
update state, bitcoin relay, package install markers, server shutdown
state). §I: federation tombstones are now flushed durably in
storage/sync so cleared peers can't resurrect after a crash.
Tracker updated with shas in docs/1.8.0-RELEASE-HARDENING-PLAN.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes from real fresh-install feedback (Framework node .81) + its log bundle:
Backend:
- websocket: subscribe before initial snapshot — broadcasts in the gap were
silently lost, stranding clients on stale state until a hard refresh
(the "everything needs ctrl-r" bug: My Apps stuck Loading, App Store
stuck Checking, containers-scanned never arriving)
- crash recovery: check the crash marker BEFORE writing our own PID —
recovery had never run on any node (always saw its own PID and skipped);
PID-reuse guard via /proc cmdline
- boot status: pending-boot-starts registry (recovery, stack recovery,
reconciler, adoption) — scanner overlays queued-but-down apps as
Restarting instead of Stopped after a reboot; scanner-authored
Restarting resolves immediately on a settled scan (no transitional wedge)
- install deps: bounded wait (36x5s) when a dependency is installed but
still starting ("Waiting for Bitcoin to start…") instead of instant
rejection; dependency-gate rejections remove the optimistic entry (no
phantom Stopped tile) and surface as a notification
- seed backup: auth.setup persists the onboarding mnemonic as the
encrypted seed backup (reveal previously failed on EVERY node — nothing
ever wrote master_seed.enc); seed.restore stashes too; error sanitizer
lets seed/2FA errors through instead of "Check server logs"
- lnd: bitcoind.rpchost resolved from the running Bitcoin variant
(hardcoded bitcoin-knots broke Core nodes); manifest uses derived_env
- bitcoin status: clean human message for connection-reset/startup; raw
URLs + os-error chains no longer reach the app card
- fedimint-clientd: chown /var/lib/archipelago/fmcd to 1000:1000 (root-
created dir crash-looped the rootless container, EACCES) — first-boot
script + pre-start self-heal
- log volume (>1GB/day on a day-old node): journald caps drop-in (ISO +
bootstrap self-heal), bitcoind -printtoconsole=0 everywhere (90% of the
journal was IBD UpdateTip spam), tracing default debug→info
Frontend:
- Login: Enter advances to confirm field then submits; submit always
clickable with inline errors (was silently disabled on mismatch);
Restart Onboarding needs a confirming second click (the mismatch →
"onboarding restarted" trap)
- sync store: 30s state reconciliation + refetch on re-entrant connect;
20s containers-scanned escape hatch so Checking can never show forever;
fresh empty node reaches the real "no apps yet" state
- intro video: CRF20 re-encode (SSIM 0.988) + faststart — moov was at EOF
so playback needed the full 15MB first (the intro lag)
- backgrounds: 10 heaviest JPEGs → WebP q90 (9.4MB→6.6MB); 7 stayed JPEG
(WebP larger on noisy sources)
- Web5ConnectedNodes: drop unused template ref that failed vue-tsc -b
ISO/kiosk:
- nginx: /assets/ 404s no longer cached immutable for a year; HTTPS block
gained the missing /assets/ location (served index.html as images)
- kiosk: launcher/service spliced from configs/ at ISO build (stale
heredoc force-disabled GPU); MemoryHigh/Max 1200/1500→2200/2800M (kiosk
rode the reclaim throttle = the lag); firmware-intel-graphics +
firmware-amd-graphics (trixie split DMC blobs out of misc-nonfree)
Verified: cargo test 898/898 green, npm run build green with dist
contents confirmed (webp refs, lnd.png, faststart video, new strings).
Handover for ISO build + deploy: docs/HANDOVER-2026-07-02-iso-feedback.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Peers that opt in via a new "Share Location" toggle in Settings
(server.set-location RPC) get plotted on other trusted peers' Mesh Map
with a distinct Archy-logo marker, separate from raw LoRa radio peers.
Location is persisted locally, carried in NodeStateSnapshot, and
propagated through federation sync/delta like other node state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- reticulum.rs: send_text_msg was lossy-UTF8-mangling binary CBOR control
envelopes (ReadReceipt etc.) before sending as LXMF text; base64-encode
with a marker instead, decoded losslessly on receive.
- typed_messages.rs: mesh.send-read-receipt fired automatically on every
chat view with no is_archy_peer gate, so viewing a message from a stock
(non-archy) LXMF peer auto-sent it an undecodable control envelope,
surfacing as garbage text right after whatever it just sent. Now a no-op
for non-archy peers.
- mesh/listener/mod.rs: RX_STALL_TIMEOUT was 300s and forced a full
auto-detect reconnect on any otherwise-healthy but quiet mesh link
(visible as "Connecting..." flapping); this also wiped Reticulum's
in-memory peer-address table every cycle, breaking messaging with peers
who hadn't re-announced in the window. Bumped to 1800s.
- reticulum.rs: persist the peer prefix/dest-hash/display-name table to
disk so a restart doesn't force every peer back to "Anonymous Peer"
until they re-announce.
- decode.rs/frames.rs: Meshcore was discarding the SNR its wire format
carries; wire it onto the peer record. Mesh.vue's signalBars() now falls
back to SNR-based bars when RSSI is unavailable (always true for
Meshcore); Reticulum has neither and correctly stays at 0/"no data".
- system/handlers.rs, dispatcher.rs: new system.get-hostname RPC + cert
regeneration (with a proper SAN) whenever server.set-name changes the
hostname, so HTTPS doesn't add a mismatch warning on top of the
self-signed one after a rename.
- AccountInfoSection.vue: surface the mDNS hostname + http/https links in
Settings (HTTPS needed for mic/camera secure-context features) — never
forced, both keep working.
- build-auto-installer-iso.sh: ship avahi-daemon so .local names actually
resolve on the LAN, and give the self-signed cert a real SAN instead of
a bare CN, both at image-build and install-time-fallback.
- Mesh.vue/MediaLightbox.vue/mesh-styles.css: mic/attach-stack no longer
closes on a plain hover-past; mesh images open in the shared lightbox
and have a real download button; lightbox close button moves to
bottom-center on mobile instead of under the status bar; mesh device
panel gets the same height/padding as its sibling tabs.
Verified: 108/108 mesh unit tests, deployed + confirmed healthy on
.116/.198/.228 (matching binary hash across all three), live Reticulum
messaging confirmed working end-to-end post-deploy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add a TollGate row (Enabled/Disabled/Not installed) to the Home
dashboard's Network tile, polling the existing openwrt.get-status RPC
on the same cadence as the other network rows. Only rendered once an
OpenWrt router is actually configured, so nodes without one aren't
cluttered with an always-"Not configured" row.
Also fixes the underlying reason this could never have worked: nothing
in the OpenWrt Gateway flow ever persisted the router's host/credentials
server-side — the "connect" form only kept them in local component
state, so any no-args openwrt.get-status call (this new tile, and even
the Gateway page's own reload) always failed with "No router
configured" despite a fully working, provisioned router. Now
handle_openwrt_get_status saves the connection to router_config.json
whenever a host is explicitly passed in and the connection succeeds.
orchestrator_uninstall_app_ids("immich") only disabled the "immich" app_id
itself; "immich-postgres" and "immich-redis" (separate orchestrator-tracked
manifests, same pattern as mempool-api/archy-mempool-db) stayed enabled, so
the boot reconciler kept restarting their leftover stopped containers
forever after the generic uninstall path stopped them (.198, 2026-07-01 --
found while uninstalling immich to relieve disk I/O pressure competing with
a slow Bitcoin IBD).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Several compounding bugs were blocking end-to-end TollGate provisioning
on OpenWrt 25.x (apk-native) routers:
- install_ipk's non-ar fallback assumed a flat tarball, but some .ipks are
a gzip tar of the three classic ipk members one level deep; it was
dumping debian-binary/data.tar.gz/control.tar.gz straight into / instead
of unpacking the real payload.
- Manually-extracted packages never ran their pending /etc/uci-defaults/*
scripts (that only happens through opkg/apk's own postinst bookkeeping),
so nothing ever created /etc/config/tollgate.
- uci_apply() never ensured the target config file existed first — `uci
set` fails outright on a config namespace nothing has created yet, which
is true for a package-defined one like "tollgate" (unlike wireless/
network/dhcp, which ship by default).
- The installed-check and restart_services looked for a binary/init script
named after the opkg package ("tollgate-module-basic-go"/"tollgate"),
but the real on-disk names are tollgate-wrt — so status always reported
"not installed" and service restarts silently no-op'd.
- provision_ssid used `uci add`, creating a new wifi-iface section (and
therefore a new duplicate broadcast SSID) on every provision call instead
of updating one in place.
Also adds a TollGateConfig.enabled field so the enable/disable state is
actually applied to the running service and the SSID's own broadcast
(stop + disable at boot, or start + enable), not just written to UCI.
On the frontend, the OpenWrt Gateway page's TollGate panel was read-only
once installed — add an edit form (price, step size, min steps, mint URL,
enabled toggle) that reuses the same idempotent provision-tollgate call.
- mempool-api now declares dependencies:[bitcoin:archival] directly, closing a
gap where installing it standalone (a legitimate direct orchestrator-install
target) bypassed the mempool umbrella's pruning gate entirely.
- New durable user-uninstalled marker (crash_recovery.rs, mirrors user_stopped)
fixes required-baseline-app self-heal (bitcoin-knots/electrumx/lnd/mempool/
etc.) resurrecting itself after an explicit uninstall survives a restart or
reboot, since the in-memory disabled set is wiped by every load_manifests().
- installed_version() (set_config.rs) no longer trusts a floating image tag
("latest") as the reported running version -- a stale local :latest cache
reported "latest" forever regardless of what latest had moved on to. Now
falls back to asking the Bitcoin backend directly via `bitcoind --version`
when the tag is floating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Master-plan backlog §10b/§10c: replace two per-app-hardcoded lookups with
generic, manifest-driven behavior so future apps are covered automatically
instead of needing a code edit.
- extract_lan_address (docker_packages.rs) now skips container-side ports
that are known non-HTTP (SSH, FTP, common DB ports) instead of blindly
taking podman's first-listed port. Fixes the whole class of bug the gitea
SSH-before-web static override was a one-off patch for.
- requires_unpruned_bitcoin (dependencies.rs) now checks the app's own
manifest for a `bitcoin:archival` dependency declaration first, falling
back to the old hardcoded id list. electrumx and mempool manifests now
declare it explicitly as the proof case.
869/869 Rust tests green, catalog drift clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merges in the meshtastic agent's now-finished work alongside this session's
continuation: stock-peer (3ccc) PKI-capability is now stamped through
get_contacts -> refresh_contacts -> MeshPeer.pkc_capable, so a directed DM to/from
a PKC-capable stock Meshtastic peer correctly shows the E2E pill on the Sent row,
not just received messages. Confirmed live: .198 sees "Meshtastic 3ccc" with
pkc_capable=true.
Also fixes two real interop/correctness bugs found while live-testing the
Reticulum <-> Sideband link:
- Receive: the daemon only ever read LXMF's plain-text content, silently
dropping native FIELD_IMAGE/FIELD_FILE_ATTACHMENTS fields — a stock
Sideband/NomadNet photo vanished into a blank-space message. Now decoded
into the same ContentInline typed envelope our own attachments use.
- Send: images to a non-archy (stock) peer now use native LXMF FIELD_IMAGE
instead of our own opaque CBOR wire format, which Sideband can't decode.
- Root cause of a garbled MC-chunk-fragment bug: TypedEnvelope.v/.sig (the
OUTER wrapper every message type uses) serialized raw bytes as a CBOR
array-of-integers instead of a native byte string, bloating every
message on the wire ~2-3.5x — enough to push even a tiny ReadReceipt
over the 140-byte single-frame chunking threshold. Root-caused by
reading ciborium's deserializer source directly (deserialize_bytes only
works within its internal scratch buffer; deserialize_byte_buf streams
unbounded).
Frontend: consolidated the attach/record buttons into a single animated "+"
menu (was overflowing the compose row).
857/857 tests pass. Verified live across all 5 deploy-roster nodes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 0 gates #2/#3 (two-node LXMF-over-LoRa, external Sideband interop) passed
on real hardware (.116's flashed Heltec V3 RNode <-> a phone-flashed RNode running
Sideband) — RNS announce, encrypted DM round-trip, and contact binding all verified
live. Fixed two bugs found in the process: the Reticulum send path wasn't stamping
outbound messages as E2E despite LXMF being unconditionally encrypted, and the
per-message transport pill collapsed Meshcore/Meshtastic into one generic "lora"
color instead of distinguishing the three radio transports.
Built on top of that link: a Columba-style image/file send experience —
compression-quality presets with a real transfer-time estimate (mesh.transport-advice,
now device-throughput-aware), receive-side thumbnail previews + auto-render for
already-local attachments, and async voice messages, all reusing the existing
ContentRef/ContentInline attachment pipeline. The headline addition is genuine RNS
Resource transfer support (daemon-side RNS.Link + RNS.Resource, Rust-side
send_resource/resource_recv plumbing, a new "resource-mesh" transport-advice tier)
so compressed photos up to 2MB now actually transfer over LoRa for Reticulum peers
instead of always falling back to Tor past the small inline-chunk cap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- WISP wizard: step-by-step flow for WiFi, DHCP, masquerade config
- WAN status: expose lan_ip, dhcp_start/limit, masq, sta_state, wifi_log
- wifi_scan: detect CCMP as WPA2 (psk2) so association succeeds
- opkg: PkgManager enum — detect apk-native mode when opkg not in repos
- tollgate: apk-native install path using manual ipk extraction
- arch detection: read DISTRIB_ARCH from /etc/openwrt_release; normalise
bare mipsel/mips from uname -m to mipsel_24kc/mips_24kc
- install_ipk: install binutils via apk when ar not in BusyBox
- install_ipk: wget --no-check-certificate for routers without CA bundle
- install_ipk: ar fallback to tar -xzf for non-standard ipk formats
- install_ipk: 5MB overlay space check with clear user-facing error
- middleware: allow "Not enough flash/space" errors through sanitizer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On a freshly-flashed OpenWrt router, radio0 is disabled by default so
iw dev returns empty. Detect the PHY via /sys/class/ieee80211/, enable
radio0, run `wifi up`, then poll up to 8s for netifd to create the
virtual interface before handing it to iwinfo scan.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New RPC methods:
- openwrt.scan-wifi: triggers iwinfo scan on the router radio,
returns networks sorted by signal strength
- openwrt.configure-wan: creates UCI wireless.wwan (sta mode) +
network.wwan (DHCP) + adds wwan to firewall WAN zone, then
calls `wifi reload`
get-status now includes a `wan` object with configured/ssid/ip/
internet fields so the UI can show current uplink state.
Frontend WAN panel: scan → pick SSID (signal bars) → enter password
→ apply. Shows "Configure WAN first" hint above TollGate install
button when internet is not available.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
apk errors were being silently dropped (stdout only). Run apk update
first and fail with a clear "router may have no internet" message if
it fails, rather than a cryptic exit-1 from apk add.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
"opkg not found at /usr/bin/opkg" was being swallowed by the error
sanitizer and shown as generic "Operation failed". Also fix bare
`opkg list-installed` call in get-status handler to use full path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BusyBox opkg exits 0 even when 'Cannot install' due to insufficient space,
causing the fallback to silently report success. Now captures stderr and
checks for the failure string explicitly.
Adds user-visible error for the common case where the router flash is too
small for the TollGate package (~19 MB needed vs ~9 MB available on typical
budget routers). Adds error prefixes to the RPC sanitizer allowlist so the
message reaches the UI instead of showing 'Check server logs'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- OpenWrtGateway.vue: add "Install TollGate" button when not installed;
tracks connected credentials for reuse in the provision call
- install.rs: fall back to wget download from GitHub releases when the
package is not in any opkg feed (mips_24kc and other arches supported)
- openwrt.rs: provision-tollgate now falls back to saved router_config
for credentials, matching the behaviour of get-status
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without these prefixes in the allowlist, sanitize_error_message swallowed
the "No router configured" error and returned a generic "Operation failed",
so the frontend could never detect the unconfigured state and show the
connect form.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend: new `openwrt.get-status` RPC endpoint SSHes into the saved (or
provided) OpenWrt router and returns system info, TollGate config, and WiFi
AP interfaces via UCI.
Frontend: new OpenWrtGateway.vue view at /dashboard/server/openwrt shows
system hostname, OpenWrt version, uptime, TollGate install/enable state with
pricing and mint URL, and all AP-mode WiFi interfaces. Linked from the Local
Network section of the Server view.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New `archipelago-openwrt` workspace crate provides SSH/UCI-based management
of OpenWrt routers, including automated TollGate installation and configuration
of a pay-as-you-go "archipelago" SSID backed by the local Cashu mint.
Exposes two RPC endpoints:
- `openwrt.scan` — discover OpenWrt routers on the LAN
- `openwrt.provision-tollgate` — install tollgate-module-basic-go, write UCI
config (TIP-01/TIP-02), and create isolated WiFi SSID + firewall zone
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- send_message now sends archy↔archy plain text as a native TEXT_MESSAGE_APP
DM (firmware PKC-encrypts E2E), not wrapped in the binary typed envelope
that silently broke archy↔archy LoRa delivery. Archy peers' Sent rows are
marked encrypted so the E2E pill shows; rich typed msgs still use the
typed-wire path.
- Add a software radio-reboot to recover a wedged/RX-deaf radio without
physical access (and for the Device-tab settings panel): driver reboot()
via AdminMessage reboot_seconds=97 (verified vs meshtastic/protobufs),
MeshCommand::RebootRadio, MeshService::reboot_radio, RPC mesh.reboot-radio.
- Handoff doc: docs/SESSION-1.8.0-OTA-PROGRESS.md "RESUME HERE" — RF link is
the proven blocker (radios not hearing each other); modem_preset mismatch
is the prime suspect; on-device Meshtastic-app check + fix plan documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
netbird is fully manifest-driven (apps/netbird-*/manifest.yml via the signed
catalog): install_stack_via_orchestrator renders the 3-member stack with
generated_certs (self-signed TLS for the #15 OIDC secure context), base64
generated_secrets, and templated config — and adopts the running stack by live
container name. The hardcoded `podman run` fallback was therefore dead code on
any node with the embedded catalog (verified live: .228 https:8087 -> 200).
Removes the per-app Rust installer anti-pattern the master plan calls out:
- install_netbird_stack: orchestrator -> adopt -> bail! (no in-Rust installer)
- deletes 6 now-dead helpers (write_netbird_config_files, ensure_netbird_tls_cert,
read_or_generate_b64_secret, netbird_net_resolver_ip, detect_netbird_public_host_ip,
wait_for_netbird_oidc_ready), 3 NETBIRD_*_IMAGE consts, unused base64::Engine import
- ~485 lines removed; prod_orchestrator doc-comments updated
Behavioural parity: the manifest path already executed on the fleet, so this
changes no live behavior. The legacy #10 OIDC-readiness wait was already bypassed
by the manifest path; if that race resurfaces, add an OIDC-ready gate to the
manifest rather than resurrecting the Rust fn.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
handle_package_uninstall lumped every teardown failure into one `errors` vec
and returned Err on any of them BEFORE removing the package state entry — so a
non-fatal cleanup hiccup (a slow/failed `sudo rm -rf` of a large data dir, a
volume/network removal) left the app's containers gone but its entry in
package_data → a ghost in My Apps, and the spawned task reverted it to Installed.
Split the failures: container removal that even force-rm can't complete (app
genuinely still present) keeps the entry + returns Err; everything after the
containers are gone is best-effort. Remove the state entry as soon as the
containers are gone — BEFORE the slow volume/data teardown — so My Apps updates
immediately and residue can never ghost the app. set_uninstall_stage is a no-op
once the entry is gone (if-let guard), so the later stages don't re-create it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
package.restart resolved its container list via
ordered_containers_for_start, which injected every name from the
union startup_order list that wasn't already present — including
variant names not live on a given node (mysql-mempool,
archy-mempool-api, archy-mempool-web). The phantom mysql-mempool is
2nd in the mempool start order, so do_orchestrator_package_start hit
its unknown-app-id fallback, do_package_start failed the inspect
("no such object"), and the `?` aborted the whole start sequence —
leaving mempool-api + the frontend down until the health monitor
recovered them minutes later. That was the source of the 5× gate
flakes #73 (frontend not running in 180s) and #74 (api not queryable
in 300s); root-caused from the .228 journal
("Start failed: mysql-mempool").
Replace the inject-then-sort logic with a pure helper
order_present_containers that orders only the actually-present
containers and never adds phantom entries. startup_order remains a
union of name variants across install generations — it's now used
purely to order what's live, not to inject what isn't. +3 unit tests.
Also harden bitcoin-knots.bats "valid state" probe: poll ≤30s for a
settled state instead of a single-shot read, so a container caught
mid-reconcile (transient restarting/configured) can't flake a 20-min
iteration. A genuinely-stuck container never settles, so real
breakage is still caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A user-stopped backend (electrumx, bitcoin, lnd, fedimint) kept reading 'running'
in container-list because its UI companion (electrs-ui, …) still serves the launch
port, and the state-refresh upgrades any reachable launch port to 'running'. The
gate's wait_for_container_status <app> stopped therefore never saw 'stopped'.
Fix: load the user_stopped marker in handle_container_list and force 'stopped' for
those apps before the launch-port refresh. The reconcile guard keeps the backend
down, so the marker is authoritative. package.start clears it first, so a started
app reports 'running' normally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author the IndeedHub stack as 7 manifests (postgres/redis/minio/relay/api/
ffmpeg + frontend) and route install_indeedhub_stack through the
orchestrator first (immich pattern), falling back to the legacy installer
only when the manifests aren't deployed.
Data-preserving by construction — the manifests reproduce the live install
exactly so an existing node ADOPTS rather than recreates:
- container_name = the live hyphenated names the runtime already references
(health_monitor tiers/deps, crash_recovery).
- named volumes indeedhub-{postgres,redis,minio,relay}-data (not bind mounts).
- dedicated indeedhub-net + network_aliases [postgres|redis|minio|relay|api]
so the api/ffmpeg env hostnames and the frontend nginx upstreams resolve
unchanged.
- generated_secrets (indeedhub-db-password/-minio-password owned by their
backends, indeedhub-jwt by the api) reuse the live /var/lib/archipelago/
secrets values (ensure_one no-ops on existing files; postgres pw is fixed
at PGDATA init). minio user "indeeadmin" + AES_MASTER_SECRET literal kept.
The frontend carries the post_install hook (#20) that replaces the hardcoded
patch_indeedhub_nostr_provider: strip X-Frame-Options, refresh
nostr-provider.js from /opt/archipelago/web-ui, inject the <script> if
absent, reload nginx — defensive/idempotent since indeedhub:1.0.0 already
bakes these. Frontend manifest also corrected off its dead Next.js shape
(health check now nginx :7777, tmpfs /run + /var/cache/nginx).
Builds + unit-tested; live adoption/lifecycle verification on .228 next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After the manifest migration the launcher installed as "immich-server" (app_id),
which has no catalog entry → showed the raw id and no icon. Rename the server
manifest app_id immich-server→immich so it matches the catalog/curated "immich"
entry (title "Immich", icon immich.png) and is recognised as a known launcher app
(APP_CATEGORY_MAP) → stays in My Apps. immich_stack_app_ids now installs
[immich-postgres, immich-redis, immich]; orchestrator.install bypasses package
routing so there's no recursion with the "immich"→stack-installer mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
