play() on the underlying <audio> element rejects independently of its
'error' event (e.g. NotSupportedError when a peer-content request 404s and
there's no decodable source) — the 'error' listener already sets a friendly
message, but the unawaited play() promise still surfaced as a raw unhandled
rejection in the console. Follow-up from the .116->.228 peer-content
investigation (2026-07-01).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
orchestrator_uninstall_app_ids("immich") only disabled the "immich" app_id
itself; "immich-postgres" and "immich-redis" (separate orchestrator-tracked
manifests, same pattern as mempool-api/archy-mempool-db) stayed enabled, so
the boot reconciler kept restarting their leftover stopped containers
forever after the generic uninstall path stopped them (.198, 2026-07-01 --
found while uninstalling immich to relieve disk I/O pressure competing with
a slow Bitcoin IBD).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- prod_orchestrator.rs: the boot reconciler's zombie-guard and start-failed
recreate paths (Created/Stopped/Exited states) had no attempt cap, unlike
health_monitor's independent restart tracker. A container whose entrypoint
fatally crashes right after `podman start` succeeds got stop+remove+
install_fresh'd every ~30s reconcile tick forever (portainer on .198,
2026-07-01: a DB schema newer than the pinned binary could read -- no
amount of recreating fixes that). Added a 5-attempts/30-minute circuit
breaker; once exhausted the container is left alone with an error! log
instead of looping, and an explicit install/start clears the counter.
- content_server.rs: serve_content now prunes a catalog entry whose backing
file is missing on disk, instead of leaving it advertised to every peer
forever with no way to distinguish "gone" from "transient failure."
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- crash_recovery.rs: stack boot/runtime recovery (immich/indeedhub/netbird) now
requires the stack's core dependency container to exist before touching any
sibling, instead of firing on any leftover container. Fixes an infinite
120s-interval crash loop where orphan debris from a partial/failed install
(indeedhub-api with no indeedhub-postgres ever created) was repeatedly
force-restarted against a dependency that doesn't exist, which also blocked
a real reinstall via container name conflicts.
- AppSessionFrame.vue: the generic app-loading overlay and the ElectrumX
sync-in-progress overlay could render simultaneously (same z-index) during
launch. The sync screen is strictly more informative, so it now takes
precedence instead of the two stacking on top of each other.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- mempool-api now declares dependencies:[bitcoin:archival] directly, closing a
gap where installing it standalone (a legitimate direct orchestrator-install
target) bypassed the mempool umbrella's pruning gate entirely.
- New durable user-uninstalled marker (crash_recovery.rs, mirrors user_stopped)
fixes required-baseline-app self-heal (bitcoin-knots/electrumx/lnd/mempool/
etc.) resurrecting itself after an explicit uninstall survives a restart or
reboot, since the in-memory disabled set is wiped by every load_manifests().
- installed_version() (set_config.rs) no longer trusts a floating image tag
("latest") as the reported running version -- a stale local :latest cache
reported "latest" forever regardless of what latest had moved on to. Now
falls back to asking the Bitcoin backend directly via `bitcoind --version`
when the tag is floating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PyInstaller-based daemons (e.g. the Reticulum mesh daemon) self-extract
to /tmp on every launch and leak their extraction dir on abnormal exit;
combined with release-build staging dirs this exhausted the default cap
on .116 and silently broke Reticulum ("no space left on device" during
self-extraction, masquerading as a connect failure). Ship a tmp.mount.d
override (75% of RAM) in the installer image so fresh installs don't
inherit the same ceiling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Master-plan backlog §10b/§10c: replace two per-app-hardcoded lookups with
generic, manifest-driven behavior so future apps are covered automatically
instead of needing a code edit.
- extract_lan_address (docker_packages.rs) now skips container-side ports
that are known non-HTTP (SSH, FTP, common DB ports) instead of blindly
taking podman's first-listed port. Fixes the whole class of bug the gitea
SSH-before-web static override was a one-off patch for.
- requires_unpruned_bitcoin (dependencies.rs) now checks the app's own
manifest for a `bitcoin:archival` dependency declaration first, falling
back to the old hardcoded id list. electrumx and mempool manifests now
declare it explicitly as the proof case.
869/869 Rust tests green, catalog drift clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Guided prompt that pops up when a mesh radio is detected but not yet
connected -- wraps the existing Connect action (mesh.configure with
device_path) rather than building a new setup engine. Dismissible per
device path (won't re-prompt for the same undismissed-but-ignored device on
every poll tick). Not the whole-app identity/seed onboarding system
(useOnboarding.ts) -- confirmed unrelated, this is mesh-specific only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three fixes:
1. Modem-preset authoritative: parse_config_lora_region now also decodes
modem_preset (field 2) alongside region, tracked as current_modem_preset.
ensure_lora_region's "region already set, don't touch it" branch (correct,
unchanged) now ALSO re-asserts LONG_FAST when a real observed preset has
drifted -- previously modem_preset only ever got written when region was
UNSET, so a radio with the right region but wrong preset was never fixed.
Only acts on an actually-observed wrong value (never speculative), so it
can't reboot-loop.
2. RX-stall watchdog: run_mesh_session now bails (triggering the existing
auto-reconnect path) if no frame has been successfully received in 5
minutes -- the existing consecutive_write_failures counter is blind to a
receive-only stall (writes can keep succeeding while inbound streaming is
wedged).
3. Hot-swap detection: spawn_mesh_listener now compares self_node_id across
session restarts and logs clearly when the physical radio itself changed
(not just an ordinary reconnect of the same board). Per-session device
state (contacts, current_region, etc.) was already naturally isolated
per-session (fresh struct each reconnect) -- nothing else needed clearing.
107/107 mesh tests pass (2 new: modem_preset decode + the
absent-field-defaults-to-LONG_FAST case).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New MeshDevicePanel.vue, added as a 4th/5th tab entry to activeTab/toolsTab/
mobileTab following the exact existing pattern (chat/bitcoin/deadman/
assistant/map). Shows firmware version, node ID, advert name, LoRa region,
channel, and device type -- firmware_version/self_node_id were already
server-side but never rendered; region is new (composed into MeshStatus from
MeshConfig.lora_region at read time, not part of the live session state).
Reboot button wired to the already-working mesh.reboot-radio RPC.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bars UI (signalBars/.mesh-signal-bars) was already built and wired to
mp.primary_rssi -- it just needed real backend data, which the previous
commit provides. Adds primary_snr alongside primary_rssi in MergedPeer and a
hover tooltip showing exact dBm/SNR values.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Backend: parse_mesh_packet now decodes MeshPacket.rx_snr (field 8, float) and
rx_rssi (field 12, int32), and a new POSITION_APP branch decodes Position.
latitude_i/longitude_i (fields 1/2, sfixed32) -- all field numbers confirmed
against the canonical meshtastic/protobufs mesh.proto, not guessed. Threaded
through ParsedContact -> refresh_contacts -> MeshPeer (mirroring how
pkc_capable was wired for #17), so mesh.peers now surfaces real rssi/snr/lat/
lon instead of always-null. Fixed a real bug found along the way:
update_node_info's unconditional contact replace would have silently wiped
any already-tracked signal/position data on the next NodeInfo packet -- now
preserves it.
Frontend: mesh.ts's updateNodePositionsFromPeers() feeds real position data
into the SAME nodePositions map MeshMap.vue already renders from (parallel to
the existing Coordinate/Alert-message path) -- MeshMap.vue itself needed zero
changes, it was already built for this.
105/105 mesh tests pass (4 new: rx_snr/rx_rssi decode, position decode +
incomplete-field handling, full packet_to_inbound_frame integration).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Field numbers confirmed against the canonical meshtastic/protobufs mesh.proto
(rx_snr=8 float, rx_rssi=12 int32), not guessed. Not yet threaded through to
ParsedContact/MeshPeer/mesh.peers — that's the next step. Part of the
Meshtastic 1.8.0 backlog plan (RSSI/SNR indicator, peer-location map, Device
tab, provisioning robustness, onboarding modal) — see
.claude/plans/floofy-riding-seahorse.md for the full plan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merges in the meshtastic agent's now-finished work alongside this session's
continuation: stock-peer (3ccc) PKI-capability is now stamped through
get_contacts -> refresh_contacts -> MeshPeer.pkc_capable, so a directed DM to/from
a PKC-capable stock Meshtastic peer correctly shows the E2E pill on the Sent row,
not just received messages. Confirmed live: .198 sees "Meshtastic 3ccc" with
pkc_capable=true.
Also fixes two real interop/correctness bugs found while live-testing the
Reticulum <-> Sideband link:
- Receive: the daemon only ever read LXMF's plain-text content, silently
dropping native FIELD_IMAGE/FIELD_FILE_ATTACHMENTS fields — a stock
Sideband/NomadNet photo vanished into a blank-space message. Now decoded
into the same ContentInline typed envelope our own attachments use.
- Send: images to a non-archy (stock) peer now use native LXMF FIELD_IMAGE
instead of our own opaque CBOR wire format, which Sideband can't decode.
- Root cause of a garbled MC-chunk-fragment bug: TypedEnvelope.v/.sig (the
OUTER wrapper every message type uses) serialized raw bytes as a CBOR
array-of-integers instead of a native byte string, bloating every
message on the wire ~2-3.5x — enough to push even a tiny ReadReceipt
over the 140-byte single-frame chunking threshold. Root-caused by
reading ciborium's deserializer source directly (deserialize_bytes only
works within its internal scratch buffer; deserialize_byte_buf streams
unbounded).
Frontend: consolidated the attach/record buttons into a single animated "+"
menu (was overflowing the compose row).
857/857 tests pass. Verified live across all 5 deploy-roster nodes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Phase 0 gates #2/#3 (two-node LXMF-over-LoRa, external Sideband interop) passed
on real hardware (.116's flashed Heltec V3 RNode <-> a phone-flashed RNode running
Sideband) — RNS announce, encrypted DM round-trip, and contact binding all verified
live. Fixed two bugs found in the process: the Reticulum send path wasn't stamping
outbound messages as E2E despite LXMF being unconditionally encrypted, and the
per-message transport pill collapsed Meshcore/Meshtastic into one generic "lora"
color instead of distinguishing the three radio transports.
Built on top of that link: a Columba-style image/file send experience —
compression-quality presets with a real transfer-time estimate (mesh.transport-advice,
now device-throughput-aware), receive-side thumbnail previews + auto-render for
already-local attachments, and async voice messages, all reusing the existing
ContentRef/ContentInline attachment pipeline. The headline addition is genuine RNS
Resource transfer support (daemon-side RNS.Link + RNS.Resource, Rust-side
send_resource/resource_recv plumbing, a new "resource-mesh" transport-advice tier)
so compressed photos up to 2MB now actually transfer over LoRa for Reticulum peers
instead of always falling back to Tor past the small inline-chunk cap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Inbound Meshtastic text addressed to BROADCAST_NUM (the default public
LongFast channel, or any channel slot) was filed into a per-sender 1:1 DM
thread, so public-channel messages polluted individual people's DM chats
and appeared as if sent directly to the user.
packet_to_inbound_frame now detects `to == BROADCAST_NUM` and emits a new
synthetic RESP_MESHTASTIC_CHANNEL_TEXT frame
([channel_idx][sender_prefix(6)][text]) that the listener files under the
channel thread (contact_id = u32::MAX - idx) while still attributing the
message to its real sender. Directed text (to == our node) still routes to
the DM thread — a regression test locks that split in.
send_channel_text now sets MeshPacket.channel (field 3) so archy actually
transmits on channel 0 (public) instead of ignoring the slot. Mesh.vue keeps
the synthetic "Meshtastic !xxxx" sender id when that is the best identity
available for a stock public-channel device.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Received messages snapshot peer_name at receive time, so a Meshtastic
text that arrived before its sender's NodeInfo was stuck showing the
synthetic "Meshtastic !xxxx" id forever, and channel/group bubbles
showed no sender at all. Add a per-bubble sender label for received
messages in multi-sender views (mesh + Archipelago channels), resolved
LIVE from the peer table so it always shows the current archy identity
(e.g. "Arch Optiplex") the moment NodeInfo is learned. Falls back to
"Unknown sender" rather than echoing a Channel/synthetic placeholder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
archy went deaf to inbound LoRa packets after every config write.
A config write (region/channel/owner) reboots the radio, which resets
the firmware PhoneAPI to STATE_SEND_NOTHING; it won't stream received
packets again until the client re-sends want_config. archy ignored
FromRadio.rebooted (field 8) so never resubscribed — which is why old
messages only arrived after a full restart (restart = fresh want_config).
- meshtastic.rs: handle FROM_RADIO_REBOOTED -> set pending_reinit;
try_recv_frame re-sends want_config to resubscribe the packet stream.
Add send_keepalive (bare heartbeat) and pin modem_preset=LONG_FAST in
set_lora_region so all radios share frequency.
- listener/session.rs: MeshRadioDevice::send_keepalive; 10s sync_timer
sends a keepalive each tick (insurance vs 15-min idle serial close).
- mod.rs send_message: device-aware send — Meshtastic archy peers get a
plain TEXT_MESSAGE_APP DM (firmware PKC E2E); Meshcore archy peers keep
the typed envelope (no meshcore regression).
Verified: .198->.228 directed DM arrives as RECEIVED enc=True
peer="Arch Optiplex"; all 3 nodes (.116/.198/.228) + 3ccc hear each
other. Binary 737b16c3 deployed+active on all three.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- send_message now sends archy↔archy plain text as a native TEXT_MESSAGE_APP
DM (firmware PKC-encrypts E2E), not wrapped in the binary typed envelope
that silently broke archy↔archy LoRa delivery. Archy peers' Sent rows are
marked encrypted so the E2E pill shows; rich typed msgs still use the
typed-wire path.
- Add a software radio-reboot to recover a wedged/RX-deaf radio without
physical access (and for the Device-tab settings panel): driver reboot()
via AdminMessage reboot_seconds=97 (verified vs meshtastic/protobufs),
MeshCommand::RebootRadio, MeshService::reboot_radio, RPC mesh.reboot-radio.
- Handoff doc: docs/SESSION-1.8.0-OTA-PROGRESS.md "RESUME HERE" — RF link is
the proven blocker (radios not hearing each other); modem_preset mismatch
is the prime suspect; on-device Meshtastic-app check + fix plan documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ensure_lora_region previously force-overrode the device's region with the
mesh-config region (EU_868) whenever they differed — which would shove a US/ANZ
user's radio onto EU_868: an illegal band that also cuts it off from its local
mesh. Off-the-shelf interop must respect whatever region the user flashed.
Now: a radio that already reports a REAL region (US, EU_868, ANZ, …) is left
untouched. We only set a region when the device reports UNSET (a fresh radio is
RF-silent and can't mesh at all), using the operator-configured region as the
fallback. Unknown/None (never reported) is also left alone. Pairs with the
default-channel change so a meshtastic archy node behaves like a stock device.
cargo check green (built into the same binary as the channel fix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make a meshtastic-equipped archy node work like a stock Meshtastic device AND
keep the private archy group, instead of being isolated on a custom primary:
- slot 0 (PRIMARY) = the DEFAULT public channel (empty name + default key) →
interoperates with every off-the-shelf device on LongFast and picks up
default-channel users; our NodeInfo broadcasts ride here like normal.
- slot 1 (SECONDARY) = "archipelago" (deterministic psk) → private archy↔archy.
Previously the driver set "archipelago" as the PRIMARY, isolating archy from the
public mesh. Now ensure_channel writes at most one channel per call (default
primary first, then archipelago secondary), reusing the existing reboot→
reconnect→re-check loop so it converges in ≤2 cycles without reboot-looping;
primary_is_default() accepts the default key in 1-byte or expanded form so a
stock radio is never needlessly rewritten. set_channel generalized to
(index, name, psk, role); want_config parse tracks both slots.
MeshCore needs no change — it never overrides channels (ensure_channel is a
no-op) and already rides MeshCore's default Public channel off the shelf.
cargo check green. NEEDS radio verify on .116/.198 (default-channel RX + archy
group on the secondary). Channel provision cap (3) covers the 2-write migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mesh/federation messages between co-located nodes were always falling back to
Tor because the FIPS overlay had no direct peering — every node depended on the
global anchor's spanning tree, and when that anchor link flaps a node is
isolated and all FIPS dials time out. (Diagnosed live on .116/.198: pure-FIPS
direct peering over UDP 8668 fixes it — 2.5ms vs timeout.)
Generalize the manual fix: in the existing 5-min FIPS seed-anchor apply loop,
also auto-connect every federation peer the PeerRegistry knows both a LAN
address AND a FIPS npub for, dialing its FIPS UDP transport (port 8668) at its
LAN IP via the same idempotent `fipsctl connect` path (new
anchors::lan_fips_anchors). This is FIPS's own transport over the LAN — NOT
Tailscale, NOT the HTTP/LAN messaging port. Transient (recomputed each tick from
live mDNS discovery, never persisted) so changing IPs self-correct. Remote peers
with no LAN address are untouched (still routed via the anchor).
Registry Arc hoisted out of the transport-init block so the loop can read
all_peers(). cargo check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The synthetic meshcore-style frame the meshtastic driver builds can't carry the
radio's PKI-encryption status, so received meshtastic DMs never lit the E2E pill.
Thread it out-of-band: the device records `last_rx_encrypted` (= packet
pki_encrypted) when it yields a text frame; the session loop reads it via
`take_rx_encrypted()` right after dispatch and stamps the just-stored received
message E2E (dispatch::stamp_received_encrypted, monotonic-id keyed). Meshcore
returns false here (its E2E is derived in the frames decrypt path). Pure
out-of-band signal — no change to the shared meshcore wire format.
Built + deployed live in binary d937814e on .116/.198. cargo check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- generate-app-catalog.sh: VERSIONS map now lists the full Knots set
(29.3.knots20260508/20260507/20260210 + 29.2.knots20251110) and Core
(adds 29.2 + a `latest` entry → newest); generator forces top-level
`version` == the default entry's version (the 169ff2e2 invariant) so
regeneration is reproducible. releases/app-catalog.json regenerated.
- docs/bitcoin-version-bulletproof-rollout.md: full handoff — root causes,
fixes, current .228 state, the coordinated fleet-rollout steps (incl.
:latest repoint sequencing / fleet-safety), reindex finish procedure, and
the switch-matrix test plan.
- PRODUCTION-MASTER-PLAN.md: link the rollout doc (§6b-bis).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three stacked bugs made "switch version" silently fail / crash-loop, and
the data-access mismatch corrupted a node's index during recovery attempts.
Backend renderer:
- sync_quadlet_unit ignored the per-app pinned version and re-rendered the
quadlet with the manifest's :latest every reconcile tick, reverting any
switch. Factor the install-time catalog/pin resolution into a shared
resolve_catalog_image() and call it in BOTH install_fresh and
sync_quadlet_unit.
- The renderer folded manifest `entrypoint: ["sh","-lc"]` into Exec=, which
only worked when the image entrypoint was a passthrough shell wrapper. The
versioned images use ENTRYPOINT ["bitcoind"], so Exec=sh -lc ... became
`bitcoind sh -lc ...` and crash-looped. Emit a real Entrypoint= override;
exec_changed now also compares Entrypoint=.
Images:
- Build all bitcoin images (Core + Knots, every version) as container-root
(USER removed) like the legacy :latest image. Chain data is owned by the
data_uid (container uid 102); root reads it via CAP_DAC_OVERRIDE (granted in
the manifest). A non-root USER (the previous uid 1000) can't read existing
chain data → "Error initializing block database". Still fully rootless:
container-root maps to the unprivileged host service user.
Catalog:
- bitcoin-knots versions[]: 29.3.knots20260508/20260507/20260210 +
29.2.knots20251110, "latest" tracking newest.
- bitcoin-core versions[]: add 29.2 + a "latest" entry. All images rebuilt
root and published to the mirror.
Frontend:
- AppSidebar version dropdown: rename the latest option to "Always use the
latest version" (no v prefix), fix right padding, and guarantee the current
selection matches a real option (was rendering blank).
- New InstallVersionModal: full-screen version chooser shown from the App
Store / Discover install button for multi-version apps (Bitcoin Knots/Core),
app icon + "Install <name>", latest pre-selected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the open Meshtastic parity bug (project_meshtastic_parity): the
running driver received nothing (`mesh.messages` stayed []) though the radio
got the packets and sends worked.
Root-cause candidate: `try_recv_frame` decoded ONE serial frame per poll and
returned Ok(None) for every non-text FromRadio frame, so the session loop slept
50ms between frames. Under Meshtastic's frequent NodeInfo/telemetry stream a
received text packet queued behind them, and read_from_radio's 64KB buffer cap
could drain (drop) it before it was ever decoded — reception silently dead while
sends kept working.
- try_recv_frame now drains a bounded batch (64) per poll, processing each
frame's side effects and returning the first inbound text frame, so a text
packet is decoded the same poll it arrives and the buffer never grows enough
to hit the lossy cap. Bounded so a continuous flood still yields to select!.
- packet_to_inbound_frame logs every decoded packet (from/portnum/payload_len)
and a "did not parse (dropped)" case, so one live radio pass is conclusive.
The rest of the decode path was verified correct by inspection (FROM_RADIO_PACKET
=2, wire-type-5 handled, parse_mesh_packet sound, 60s heartbeat present) — not a
parse bug. cargo check green. NEEDS a live radio pass on a rig that isn't .228
(off-limits: bitcoin testing) to confirm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a per-message transport badge to archy↔archy mesh chats and fixes the
long-broken E2E badge — both meshcore and meshtastic, styled like the existing
E2E pill.
Transport pill:
- New `MeshMessage.transport` ("lora"/"fips"/"tor"), surfaced in the UI beside
the E2E badge (Mesh.vue transportLabel() → Mesh/FIPS/Tor, mesh-styles.css).
- Sent LoRa → "lora"; sent federation → finalized to the real leg ("fips"/"tor")
once the background send resolves (req.send_json transport), via an id-keyed
store update.
- Received: a post-dispatch stamp on handle_typed_envelope_direct's output
(monotonic ids) tags both transports without threading through all 20 typed-
dispatch sites — radio wrapper stamps "lora", federation injector stamps the
peer's last_transport ("fips"/"tor", default tor; the inbound HTTP carries no
FIPS-vs-Tor signal).
- Plain native/channel LoRa frames → "lora"; channel broadcasts stay non-E2E.
E2E pill fix:
- `encrypted` was hardcoded false at every MeshMessage construction site, so the
UI badge (Mesh.vue `v-if="msg.encrypted"`) never showed. Now: federation
envelopes are E2E (identity-signed over an encrypted transport); the meshcore
native-DM receive path already had a real `encrypted` flag (now also tagged
with transport). meshtastic-PKI radio E2E flag threading is a noted follow-up.
Backend cargo check + frontend vue-tsc build both green. Needs a live radio +
multi-transport pass on .116/.228 to confirm end-to-end (see
project_transport_pill / project_meshtastic_parity).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The knots versions[] marked 29.3.knots20260508 as default while the
top-level catalog version is the floating 'latest' tag — violating the
generator's own invariant (default:true MUST equal the top-level version
so selecting it un-pins / tracks latest). Live effect via package.versions:
catalog_default_version='latest' so the UI-highlighted default actually
PINS+recreates (opposite of un-pin) and 'latest' was unreachable from the
Version & Updates card.
Add a 'latest' default entry (== the manifest's floating tag) and keep
29.3.knots20260508 as a pinnable option. Verified on .228: package.versions
now returns default=latest with 2 selectable versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two Phase-3 bats suites used `fail` (a bats-assert helper) but bats-assert
isn't installed on the alpha fleet (only bats-core), so every tripped
assertion crashed with `fail: command not found` (status 127) instead of
reporting a real pass/fail. Define the same minimal `fail() { echo ...;
return 1; }` the other suites already use (see mempool.bats). Without this
the gates were silently non-functional.
Also rewrite the obsolete "HealthCmd= implies Notify=healthy" assertion in
use-quadlet-backends-install.bats. Phase 3.4's Notify=healthy was
deliberately reverted: gating `systemctl start` on health hung boot
reconciliation for dependency-waiting apps (fedimint idles until Bitcoin
IBD; lnd until macaroon unlock), leaving units stuck "deactivating". The
renderer now emits HealthCmd= for Podman's health state but TimeoutStartSec=0
and NO Notify=healthy (quadlet.rs render() + contains_stale_health_gate()).
The test now asserts the current invariant: no backend unit gates start on
health.
Verified on the .228 canary node (ARCHIPELAGO_USE_QUADLET_BACKENDS=1):
use-quadlet-backends-install 6/6, backend-survives-archipelago-restart 3/3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The archy-mempool-web health_check endpoint used http://localhost:8080.
Inside the frontend image, wget resolves `localhost` to ::1 (IPv6) first,
but nginx binds 0.0.0.0:8080 (IPv4) only -> the baked HealthCmd gets
"connection refused" every probe -> container is perpetually unhealthy ->
the reconciler recreates it forever (observed on .228: mempool container
re-Started every ~3 min, Health=unhealthy). Proven live: in-container
`wget http://localhost:8080/` = refused, `wget http://127.0.0.1:8080/` = OK.
Pin the probe to 127.0.0.1 so it matches nginx's IPv4 bind. Updated both
the source manifest and the embedded copy in releases/app-catalog.json
(the catalog overlay wins over the disk manifest on fleet nodes, so the
catalog copy is the one that actually reaches .228).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The split-mempool-stack guard that skips the legacy monolithic `mempool`
manifest (whose container collides with its split-stack frontend member
`archy-mempool-web`) only ran over DISK manifests. On catalog-driven nodes
(no disk manifests — e.g. the Phase-3/registry-manifest path), the legacy
`mempool` manifest arrives via the registry-catalog overlay AFTER that
guard, so both `mempool` and `archy-mempool-web` end up owning container
`mempool` and rewrite+restart each other forever ("port binding drift" /
"network alias drift" loop observed on .228, leaving mempool down).
Enforce the guard once more over the merged (disk + catalog) manifest set:
drop the `mempool` umbrella whenever all three split members are present.
Installing `mempool` assembles the split stack, so `archy-mempool-web`
owns the frontend container either way.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mempool manifest + embedded catalog declared the frontend container
port as 4080, but mempool-frontend nginx listens on 8080 (the stack
creates it as -p 4080:8080 with FRONTEND_HTTP_PORT=8080, see
api/rpc/package/stacks.rs). So every reconcile rendered the quadlet as
PublishPort=4080:4080, disagreed with the working 4080:8080 container,
and restarted it ("port binding drift" -> "host port 4080 did not become
reachable within 5s" -> "host listener disappeared; restarting") in a
perpetual loop on .228. Correcting the manifest container port to 8080
makes the rendered quadlet match reality so the drift/restart loop stops.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On NAT'd nodes that can reach the iroh federation neither directly nor
via iroh's public relays, fmcd's embedded iroh networking enters a
relay/hole-punch reconnect hot-loop that pegs its entire CPU allotment
indefinitely (observed ~1 core sustained for 4 days on a Tailscale node,
while LAN nodes that reach the guardian directly stay <3%). fmcd 0.8.0
exposes no iroh/relay knobs, so:
- fmcd-run now samples fmcd's own CPU and restarts it when it stays near
its allotment for ~15 min (a restart demonstrably clears the stuck iroh
state; real work is bursty and never flat-pegs a core for minutes).
- Lower cpu_limit 1 -> 0.25 core so a stuck instance can't starve the
node (steady-state is <3% of a core; joins are brief).
Ships as fmcd:0.8.1 (launcher-only rebuild, same fmcd binary). Bumped the
image pin + cpu_limit in the manifest, image-versions.sh, the embedded
catalog manifest (releases/app-catalog.json), and the UI catalogs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WS-F #10: a netbird reinstall that adopts a leftover running container
skipped ensure_manifest_certs, so when its data dir was wiped the self-
signed tls.crt/key were never regenerated; the next nginx.conf rewrite +
restart then died on the missing cert (proxy 502, login broken). The
Running branch of ensure_running_with_mode now calls ensure_manifest_certs
before ensure_manifest_files, mirroring prepare_for_start's certs-before-
files ordering. Idempotent: a no-op when crt+key already exist.
Live-validated on .228: deleted netbird tls.crt/key under a Running
container; reconciler regenerated a fresh CN=<host_ip> self-signed cert
(1000:1000), https :8087 = 200.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Region (EU_868) + shared channel "archipelago" auto-provisioning shipped in
8fdb45e8 and riding the rolled #9 fleet binary (0060dcd6). Discovery, RF, and
sending verified on .116+.228; the one open blocker is the running driver not
surfacing received messages. Slotted after WS-F #9–11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fresh Meshtastic radios ship region-UNSET (RF-silent) and on mismatched
channels, so nodes only ever saw themselves. Bring them to MeshCore parity
using the official Meshtastic admin API:
- Auto-provision LoRa region (set_config, AdminMessage field 34) from a new
mesh-config `lora_region` (e.g. EU_868) when the radio's region differs.
- Auto-provision a shared primary channel (set_channel, field 33) with a
PSK derived deterministically from channel_name, so every node converges on
one mesh — the parity equivalent of MeshCore's named "archipelago" channel.
- Read current region/channel from want_config; only write when different
(no reboot loop); cap attempts so a radio that won't persist can't loop.
- Active NodeInfo advert scaffolding + aggressive serial drain.
Verified on .116+.228: region+channel persist, discovery works (both see each
other as named reachable contacts), bidirectional RF + sending confirmed.
Receiving in the running driver is still under diagnosis (instrumentation added).
Also removes the unwanted `meshtastic` daemon app from the registry (it was
never meant to be a container — native driver provides system-level support):
deletes apps/meshtastic + catalog entries (app-catalog, neode-ui, releases) +
test refs. Meshtastic stays native, like MeshCore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ensure_bind_mount_dirs chowned a freshly-created no-data_uid bind dir
with --reference={immediate_parent}. For a NESTED bind source like
jellyfin's /var/lib/archipelago/jellyfin/config (or netbird's .../netbird/
data), `mkdir -p` creates the intermediate <app> dir root:root too, so
referencing the immediate parent just copied ROOT — leaving the dir
unwritable and the app EACCES-crash-looping on reinstall (found by the
all-apps-lifecycle pass: jellyfin "/config/log denied" exit 139;
netbird-server "unable to open database file"). It only ever worked for
direct children of the data root (immich).
Fix: anchor to the nearest PRE-EXISTING ancestor (the rootless data root,
owned by the service user) and chown -R the entire newly-created subtree
to it. Extracted the walk into fresh_subtree_anchor() with a unit test
covering nested / direct / second-volume cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
