archy/docs/SESSION-1.8.0-OTA-PROGRESS.md

1.8.0 OTA Session Progress

Updated: 2026-06-30

---

▶️ RESUME HERE — archy↔archy LoRa (2026-06-30 PM) — READ FIRST

Goal: archy↔archy text over Meshtastic LoRa must DELIVER and show the E2E pill, identical in off-grid and normal mode. Test bed = .116 / .198 / .228 (all EU_868). Don't touch the federation/FIPS path.

✅✅✅ SOLVED 2026-06-30 — archy↔archy LoRa WORKS (delivery + E2E pill + identity)

VERIFIED: .198→.228 directed DM → .228 row RECEIVED enc=True peer="Arch Optiplex". All three nodes (.116/.198/.228) now hear each other + stock peer 3ccc. Deployed binary 737b16c3235b active on all three. Fix source COMMITTED as a57ae388 on main (not yet pushed to gitea-vps2/origin).

THE fix (receive stream): archy ignored FromRadio.rebooted (field 8). Every config write reboots the radio → firmware PhoneAPI resets to STATE_SEND_NOTHING and stops streaming received packets until the client re-sends want_config. archy never did → went deaf to inbound (that's why old messages only arrived after a full restart = fresh want_config). Fix: handle FROM_RADIO_REBOOTED → set pending_reinit → re-send want_config; plus a 10s keepalive heartbeat (insurance vs 15-min idle serial close) and a pinned modem_preset=LONG_FAST so all radios share frequency. Combined with the earlier E2E send fix (plain TEXT_MESSAGE_APP DM, firmware PKC) this closes archy↔archy LoRa.

Open follow-ups: #A surface received msgs under archy identity in all UI views; #6 device-onboarding modal; #8 Device-tab settings panel; #7 re-verify .116 in rotation; #12 make modem_preset authoritative + hot-swap re-binding + RX-stall watchdog; #14 signal-strength (RSSI/SNR) indicator per contact (from MeshPacket rx_rssi/rx_snr); #15 map view plotting peer locations where shared (Meshtastic POSITION_APP portnum=3 lat/lon). See the resume memory project_session_resume_2026_06_30_lora.md for the full task list.

(historical) earlier TL;DR — RF-layer suspicion, now RESOLVED by the reboot-recovery fix

The archy software is correct and deployed. The blocker was at the radio/RF layer: the three radios are not hearing each other over the air at all. No amount of archy code change will fix that until the radios actually RF-link. Resume by testing the radios directly at home (Meshtastic phone app over Bluetooth) — see "DO THIS FIRST AT HOME" below. ← this turned out to be the want_config resubscribe bug above.

What is DONE and deployed (commit pending — see below)

• E2E send fix (core/archipelago/src/mesh/mod.rs send_message, ~L1542): archy↔archy plain chat text is now sent as a native TEXT_MESSAGE_APP DM (firmware PKC-encrypts it E2E), NOT wrapped in our binary typed envelope. Archy peers' Sent rows are marked encrypted=true so the pill shows. Rich typed msgs still use send_typed_wire. This was the original root-cause fix (envelope-wrapped text silently broke archy↔archy LoRa).
• NEW: software radio-reboot end-to-end, so a wedged/RX-deaf radio can be rebooted without physical access (and for the Device-tab settings panel the user requested):
  • meshtastic.rs: reboot(seconds) driver method + ADMIN_REBOOT_SECONDS_FIELD = 97 (verified vs meshtastic/protobufs admin.proto — set_owner=32/set_channel=33/set_config=34 matched our existing constants, confirming the proto read).
  • listener/mod.rs: MeshCommand::RebootRadio { seconds }.
  • listener/session.rs: device-enum reboot() dispatch (Meshtastic only) + handler arm.
  • mesh/mod.rs: MeshService::reboot_radio(seconds).
  • api/rpc/mesh/messaging.rs: handle_mesh_reboot_radio → RPC mesh.reboot-radio {seconds?} (default 2); dispatcher arm in api/rpc/dispatcher.rs.
  • cargo check passes. Built release sha ba4aed590027690d and DEPLOYED + active on .116/.198/.228. The RPC works ({"reboot":true,"seconds":2}).
  • ⚠️ Caveat: when called, archy logged "Sent Meshtastic radio reboot" but the radio did not visibly reboot afterward (no config re-stream). Either field 97 is still off, or newer firmware requires an admin session passkey even over local serial, or the USB serial stayed open through the 2s reboot so no reconnect was logged. Needs on-device verification.

The hard evidence (why "nothing works")

• Directed DM tests .198→.228 AND .116→.228 (neither path reflashed): sender logs Sent plain native DM dest=30d258436d65 part=1 total=1 and RPC returns sent:true, encrypted:true, but .228 logs nothing — packet never reaches archy from the radio.
• A raw broadcast from .198 (mesh.broadcast) was accepted by its radio but not heard by .228/.116.
• In an 8-minute window, all three nodes received 0 inbound OTA packets from any other node. Each only logs its OWN once-a-minute Broadcast Meshtastic NodeInfo advert + local TX field=11 queue-status. .228 mesh.status = messages_received:1 total.
• .198's radio is alive and transmitting NodeInfo every 60s — so it's not dead; it's that reception is broken on the receivers. A radio cannot drop a broadcast AND a unicast to its own node number while config matches, unless it simply isn't on the same airwaves.
• archy provisioning is correct & identical across nodes (read back from device): PRIMARY = public LongFast (name="" psk_len=1), SECONDARY = archipelago, region=3 (EU_868). Admin field constants verified. The send path hands the radio a correct unicast MeshPacket (to=node, want_ack, hop_limit=3, plaintext decoded for the firmware to PKC-encrypt).

PRIME SUSPECT (software-fixable) — modem-preset / frequency mismatch

archy only ever writes region + use_preset and never explicitly pins modem_preset (it parses region but not preset; set_lora_region relies on the LongFast default). If ANY radio has a non-default modem preset / frequency slot persisted (e.g. set via the Meshtastic app, or a different factory default after the .198 reflash), the radios are on different airwaves despite identical channel name + region, and archy would never correct it.

DO THIS FIRST AT HOME (decisive, ~2 min, only the user can do it)

Open the Meshtastic phone app over Bluetooth (works alongside archy's USB serial) on each of .116/.198/.228 and check:

1. Do the 3 nodes see each other in the node list (recent "heard")? → if NO, they're not RF-reaching (preset/freq/antenna/range).
2. Do all 3 show the same Modem preset (LongFast), Region (EU_868), Frequency slot, and the same PRIMARY channel? → any difference = the cause. This single test separates "archy misconfigures the radios" from "radios physically can't reach each other."

THEN — the archy fix to apply (if preset/config differs)

Make archy authoritatively write the full LoRaConfig and force re-provision so all radios converge: in core/archipelago/src/mesh/meshtastic.rs::set_lora_region (and its caller/guard ensure_lora_region ~L304), explicitly set modem_preset = LONG_FAST (0) as a field in the LoRaConfig (it's currently omitted/defaulted), and make the startup provision path rewrite LoRa config when the preset doesn't match, then reboot the radio (use the new mesh.reboot-radio). Also verify the mesh.reboot-radio actually reboots the radio on-device (the caveat above).

TEST RECIPE (works on each node)

• RPC helper used this session: a node-side rpc.sh that logs in (password ThisIsWeb54321@), grabs the csrf_token cookie, echoes it as X-CSRF-Token, and POSTs to http://127.0.0.1:5678/rpc/v1. Recreate it or run archy's RPC directly. Methods: mesh.peers, mesh.status, mesh.messages, mesh.send {contact_id,message}, mesh.broadcast, mesh.reboot-radio {seconds}.
• LoRa contact ids: .116=1135977788 (prefix 3ca5b543), .198=3677050140 (db2b551c), .228=1129894448 (prefix 30d25843), stock 3ccc=1128152268.
• Link health check (run on each node): look for inbound from=Some("!...") lines in journalctl -u archipelago that are NOT the node's own Broadcast ... NodeInfo advert. If zero across all nodes → RF link is down (the current state).
• E2E success criteria: send .198→.228, the marker appears in .228 mesh.messages as an inbound row with encrypted:true / transport:"lora", AND .116↔.228 likewise.

DEPLOY / BUILD RECIPE

• Build: from core/, CARGO_TARGET_DIR=/tmp/archy-hotfix-target CARGO_INCREMENTAL=0 cargo build --release -p archipelago --bin archipelago. (If rust-lld: undefined hidden symbol, it's incremental cache — CARGO_INCREMENTAL=0 fixes it.)
• SSH key ~/.ssh/archipelago-deploy is authorized on .116/.198/.228. SSH/UI/RPC password ThisIsWeb54321@. Per node: scp the binary, sudo systemctl stop archipelago → kill -9 $(pgrep -x archipelago) → install -m0755 to /usr/local/bin/archipelago → systemctl start archipelago. Verify by sha256sum match + systemctl is-active.
• Current deployed sha on all 3 = ba4aed590027690d (the reboot-enabled build).

Fleet state (as of 2026-06-30 PM)

• All 3 nodes on binary ba4aed59, active. Off-grid mode currently OFF (mesh_only:false).
• .198 radio was reflashed to factory firmware-heltec-v3-2.7.26 (recovered from corrupt NVS); region EU_868 persists. Its archy identity is NOT re-bound on .228 (.228 shows .198 as raw radio "Meshtastic 551c", arch_pubkey_hex absent) because .228 hasn't heard .198's identity broadcast — a downstream symptom of the dead RF link, not a separate bug.
• The radios are powered & each transmitting; they are simply not hearing each other.

Deferred UI (after LoRa works)

• Device-tab settings panel (gear/desktop) — host the "Reboot radio" button there; calls mesh.reboot-radio. Scoping done: add to the Mesh.vue actions row (mirrors Broadcast/Off-Grid buttons) + a rebootRadio() method in neode-ui/src/stores/mesh.ts. See Mesh.vue ~L1484 actions row and mesh.ts ~L373 broadcastIdentity() pattern.
• Device-onboarding modal (detect plugged-in radio).

---

Current scope:

• Preserve existing mesh work: E2E indicators, FIPS/Tor transport indicators, typed-message paths, Meshtastic region/channel provisioning, and dirty Meshtastic receive-attempt changes.
• Take over the 3ccc stock Meshtastic peer bug: LoRa text from 3ccc to Archipelago .116 does not surface in mesh.messages.
• Keep release-gate fixes already made in this session.

Local gate status so far:

• cargo test -p archipelago --bin archipelago: green, 849/849 after Meshtastic fixes.
• python3 scripts/check-app-catalog-drift.py --release --strict: green.
• npm run type-check: green.

Key changes made so far:

• Added cascade uninstall progress truthfulness assertion to tests/lifecycle/bats/cascade-uninstall.bats.
• Fixed release catalog drift filters and regenerated catalog metadata.
• Fixed invalid apps/fedimint-clientd/manifest.yml cpu_limit schema value.
• Updated stale/tight Rust tests without changing production behavior.

Remaining non-automatable / operational gates:

• Workstream B signing is blocked on the offline RELEASE_MASTER_MNEMONIC; code + runbook exist, but the publisher must pin/sign the release-root catalog.
• Phase-3 Quadlet backend rollout is implemented behind use_quadlet_backends and default-off. The gate skip-passes until explicitly enabled on a node; flipping it fleet-wide requires a coordinated flag rollout plus backend reinstall/migration verification.
• .116 read-only use-quadlet-backends-install.bats: 6/6 skip-clean; no backend .container units, so Phase-3 is not active on that node.
• Release metadata still says 1.7.99-alpha in releases/manifest.json; changelog top is v1.8.00-alpha. Cutting an actual 1.8.0 OTA requires an explicit version/manifest update.

Do not discard:

• core/archipelago/src/mesh/listener/decode.rs
• core/archipelago/src/mesh/listener/session.rs
• core/archipelago/src/mesh/meshtastic.rs

3ccc bug current hypothesis:

• The prior attempted Meshtastic fix added a hard stale-packet filter using rx_time.
• Stock Meshtastic radios without GPS/RTC can report tiny nonzero epoch values until time sync.
• That would make live 3ccc packets look older than 10 minutes and get dropped before mesh.messages.
• Current patch treats implausibly early rx_time values as unknown rather than stale.

.116 live validation after 2026-06-30 hotfix:

• .116 reachable by SSH; archipelago active; /dev/mesh-radio -> ttyUSB0 attached.
• Current canary deploy is commit b4531bb4; backend sha 4ab53e539d89679ef664401a9a57996267772fed02327abc2912c3e77543acbf; frontend bundle index-YOAeJF7w.js / Mesh-BSAo88jN.js.
• main pushed to gitea-vps2.
• RPC on .116:
  • transport.status currently reports mesh_only:false (off-grid mode is not enabled unless the user toggles it).
  • mesh.status reports Meshtastic connected: device_type:"meshtastic", self_node_id:1135977788, peer_count:13.
  • Recent .116 -> 3ccc sent rows are stored with real 2026 timestamps and transport:"lora".
• UI/backend fixes included in b4531bb4:
  • transportLabel("lora") displays LoRa.
  • mesh sends refetch messages after send so transport pills settle without browser refresh.
  • off-grid mode blocks the mesh-chat FIPS/Tor federation fallback and forces LoRa-only sends; banner text is Tor/FIPS disabled - LoRa only.
  • empty mesh-chat placeholder opacity reduced.
• Meshtastic diagnostics now identify the remaining blocker:
  • 3ccc NodeInfo is discovered: Meshtastic peer is PKC-capable (NodeInfo public_key) node=1128152268 key_len=32.
  • Bytes from stock Meshtastic text reach .116, but the custom parser rejects the packet: Meshtastic FromRadio.packet did not parse into a decoded MeshPacket len=73 head=0dcc3c3e43153ca5b5432a16df56cbed.
  • Non-text packets decode and are ignored with port numbers (portnum=3/4/5), so the serial read path is alive. Resume inside core/archipelago/src/mesh/meshtastic.rs::parse_mesh_packet.
• LoRa is therefore not fully fixed yet: stock 3ccc -> .116 text does not surface in mesh.messages, and .116 -> 3ccc still needs user-visible confirmation in the Meshtastic app.