archy/docs/SESSION-1.8.0-OTA-PROGRESS.md

# 1.8.0 OTA Session Progress

Updated: 2026-06-30

> **📋 Live day-to-day task tracker: `docs/UNIFIED-TASK-TRACKER.md`.** This doc is kept
> as the historical session-by-session log; open items were consolidated into the
> unified tracker on 2026-07-01 (several turned out already shipped — see that doc for
> current status instead of re-deriving it from the log below).

---

## ▶️▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (evening) — #17 deployed + verified on .198/.228

**#17 (3ccc / stock-peer E2E pill) is now built, deployed, and live-verified** on `.198` and
`.228` only (`.116` skipped per the hardware notice below — its radio is mid-reflash to RNode).

- Built release binary **sha `b1d695fc626a7382`** from the working tree (`cargo check` +
  `cargo test -p archipelago mesh::` both green, 99 passed/0 failed/1 ignored, right before
  building — tree was settled, no collision with the Reticulum agent's concurrent edits).
- Deployed via stop/swap/start to `.198` (192.168.1.198) and `.228` (192.168.1.228), sha256
  confirmed matching on both, `systemctl is-active` = `active` on both (`.228` took its usual
  ~couple-minute convergence — heavy resilience node, unrelated bitcoind/fedimint container
  startup noise in the logs during that window, no mesh errors).
- **Live-verified the actual fix**, not just deploy: on `.198`, `mesh.peers` shows
  `"advert_name":"Meshtastic 3ccc", "pkc_capable":true`, and `mesh.send` to 3ccc
  (`contact_id:1128152268`) now returns **`"encrypted":true`** — confirms the
  `archy || peer_pkc_capable(contact_id)` TX fix is live, not just compiled.
- `.228`'s RPC password in memory (`password123`) was stale — user confirmed the correct
  password is `ThisIsWeb54321@` (same as `.198`/`.116`, i.e. fully unified now). Re-verified via
  RPC: `mesh.peers` shows 3ccc `pkc_capable:true`, and `mesh.send` to 3ccc returns
  `"encrypted":true` — #17 confirmed live on `.228` too, not just `.198`.

**NOT yet done:** push commit to gitea-vps2 (still uncommitted in the working tree, by design —
shares the tree with the Reticulum agent's uncommitted work); user on-device confirmation that
the E2E pill actually renders in the Mesh UI for 3ccc.

---

## 🛠️ HARDWARE NOTICE 2026-06-30 (~16:30) — .116's Heltec V3 is being repurposed

**The Reticulum agent is reflashing .116's Heltec V3 (the board on `/dev/ttyUSB0`, currently
.116's live Meshtastic radio) to RNode firmware**, with explicit user approval, to unblock the
Reticulum Phase-0 hardware gates (real RNode needed; see `docs/RETICULUM-TRANSPORT-PROGRESS.md`).
This was user-confirmed specifically because it takes .116 offline as a Meshtastic radio.

**Effect on this workstream: do all on-device Meshtastic testing on .198 and .228 only — .116 no
longer has a Meshtastic-firmware radio attached once this lands.** `cargo check`/`cargo test
-p archipelago` were both confirmed clean (99/99 mesh tests) right before the reflash started, so
the earlier "wait for their edit to settle" blocker above is cleared — software-side it's safe to
build/test/deploy; only .116's *physical radio role* changed.

---

## ▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (later PM, ~15:50) — READ THIS FIRST IF RESUMING

**#17 (3ccc / stock-peer E2E pill) is CODE-COMPLETE in the working tree**, isolated
to `meshtastic.rs`/`protocol.rs`/`types.rs`/`mod.rs` as planned (no `session.rs`
transport-plumbing changes from this side):
- `ParsedContact.pkc_capable` (`protocol.rs`) + `MeshPeer.pkc_capable` (`types.rs`),
  both `#[serde(default)]`/defaulted `false` at every construction site.
- `MeshtasticDevice::get_contacts()` now stamps `pkc_capable` per contact from the
  existing `peer_is_pkc_capable(node_num)` seam (de-`allow(dead_code)`'d).
- `listener/session.rs::refresh_contacts` ORs the new value into `MeshPeer.pkc_capable`
  (capability only grows, never cleared by a transient refresh) — this IS a touch of
  session.rs, but additive/non-colliding with the Reticulum device-enum match arms
  already there; did not touch transport plumbing/routing.
- `mod.rs::MeshService::send_message` now does `archy || self.peer_pkc_capable(contact_id)`
  for the Sent-row `encrypted` flag (was `archy`-only before).
- Verified via `cargo check -p archipelago --bin archipelago` (clean, exit 0) **before**
  the other agent's latest edit landed.

**NOT YET DONE:** rebuild release binary → redeploy 5 nodes → push → user on-device test
(same as #16, both still pending live verification).

**⚠️ BLOCKED right now — do not build/deploy/push until this clears:** the Reticulum
agent is actively mid-edit in the *same* working tree. A `cargo test` run right after
the clean `cargo check` above failed with a real (but transient, not mine) signature
mismatch: `session.rs::auto_detect_and_open` / `run_mesh_session` were observed with a
new `device_kind: Option<DeviceType>` param that `listener/mod.rs`'s call site didn't
have yet — a normal in-flight snapshot of their work, not a regression to fix here.
**Action on resume: re-run `cargo check` first; if it's clean, the other agent's edit
has settled and it's safe to proceed to build/test/deploy. If still broken, wait —
do not stash, revert, or patch their in-progress session.rs/listener/mod.rs changes**
(see memory `feedback_concurrent_agent_tree.md`). Also: building/deploying right now
would bundle their not-yet-finished `reticulum.rs` wiring into the binary — confirm
with the user before shipping a combined build, since only the meshtastic `#17` piece
has been asked for/owned by this session.

---

## ▶️▶️ LIVE CHECKPOINT 2026-06-30 (late PM) — READ THIS FIRST

**Fleet state:** all **5 test nodes** on binary **`38c456b0bacec3c4`** + frontend
**`Mesh-CAkPgvLo.js`**, `archipelago` active on each:
`.116`, `.198`, `.228` (LAN, archipelago@ + `~/.ssh/archipelago-deploy`),
`100.72.136.5`, `100.89.209.89` (Tailscale, same key — installed this session;
SSH user `archipelago` / pw `ThisIsWeb54321@`; NOPASSWD sudo on all 5).

**Shipped this session (commit `12e7990b` on `main`, pushed to gitea-vps2):**
- ✅ **#16 public-channel routing** — inbound Meshtastic text to `BROADCAST_NUM`
  now files under the **public channel thread** (contact_id `u32::MAX - idx`),
  attributed to its real sender, instead of polluting per-sender DM threads.
  Directed text (`to == our node`) still routes to the DM thread (regression test
  `packet_to_inbound_frame_directed_dm_stays_a_contact_message`). `send_channel_text`
  now sets `MeshPacket.channel` so archy TX's on channel 0 (public).
  Code: `meshtastic.rs` (`packet_to_inbound_frame`, `parse_mesh_packet` to/channel,
  `send_channel_text`), `protocol.rs` (`RESP_MESHTASTIC_CHANNEL_TEXT = 0x70`),
  `listener/frames.rs` (handler + sender attribution), `Mesh.vue` (`senderLabelFor`).
  Tests green (95 mesh tests). **Pending: user on-device test with the radios.**

**Push access:** `main` is a PROTECTED branch on gitea-vps2. Direct push uses the
dedicated **`ai`** account via remote **`gitea-ai`** (`git push gitea-ai main`).
See memory `reference_gitea_ai_push_account.md`.

**Coordination:** another agent owns **Reticulum** (`reticulum-daemon/` + Rust
transport wiring). DO NOT touch `mesh/listener/session.rs` transport plumbing or
`mod.rs` routing in ways that collide. Keep #17 work isolated to `meshtastic.rs`
RX/TX + (if needed) the sent-row encrypted flag.

### ✅ CODE-COMPLETE (not yet deployed/tested live) — #17 (3ccc / stock-peer E2E pill)
Goal: DMs **to and from** a PKC-capable stock peer (3ccc, NodeInfo public_key
key_len=32 confirmed) must show the E2E pill.
- **RX side is already correct:** `parse_mesh_packet` reads `public_key` (field 16)
  + `pki_encrypted` (field 17) per the MeshPacket proto; the directed-DM RX path
  promotes to `RESP_CONTACT_MSG_V3_E2E` when `pki_encrypted`. (Verify live.)
- **TX bug (root cause) — FIXED:** `mod.rs::send_message` now records the Sent row
  with `encrypted = archy || peer_pkc_capable(contact_id)`. `peer_is_pkc_capable`
  (meshtastic.rs) is wired out via `get_contacts()` → `ParsedContact.pkc_capable` →
  `refresh_contacts` (session.rs) → `MeshPeer.pkc_capable` → `MeshService::peer_pkc_capable`.
  See the LIVE CHECKPOINT at the top of this file for the exact touch points.
- NEXT STEP when resuming: confirm `cargo check` is clean (the other agent's
  Reticulum work shares this tree and may be mid-edit — see top checkpoint), then
  rebuild → redeploy 5 nodes → push → user test (same pending step as #16).

**Remaining open after #17:** #12 (provisioning robustness — HOLD, session.rs churn
risks reticulum collision), #8 (Device-tab settings panel + reboot button — RPC
`mesh.reboot-radio` already exists), #6 (onboarding modal), #7 (.116 re-verify),
#14 (RSSI/SNR per-contact indicator), #15 (peer-location map, POSITION_APP portnum=3).

---

## ▶️ RESUME HERE — archy↔archy LoRa (2026-06-30 PM) — READ FIRST

**Goal:** archy↔archy text over Meshtastic LoRa must DELIVER and show the E2E pill,
identical in off-grid and normal mode. Test bed = `.116` / `.198` / `.228` (all EU_868).
Don't touch the federation/FIPS path.

### ✅✅✅ SOLVED 2026-06-30 — archy↔archy LoRa WORKS (delivery + E2E pill + identity)
VERIFIED: `.198→.228` directed DM → `.228` row `RECEIVED enc=True peer="Arch Optiplex"`.
All three nodes (.116/.198/.228) now hear each other + stock peer 3ccc. Deployed binary
**`737b16c3235b`** active on all three. Fix source **COMMITTED as `a57ae388`** on `main`
(not yet pushed to gitea-vps2/origin).

**THE fix (receive stream):** archy ignored `FromRadio.rebooted` (field 8). Every config
write reboots the radio → firmware PhoneAPI resets to `STATE_SEND_NOTHING` and stops
streaming received packets until the client re-sends `want_config`. archy never did →
went deaf to inbound (that's why old messages only arrived after a full restart = fresh
want_config). Fix: handle `FROM_RADIO_REBOOTED` → set `pending_reinit` → re-send
want_config; plus a 10s keepalive heartbeat (insurance vs 15-min idle serial close) and
a pinned `modem_preset=LONG_FAST` so all radios share frequency. Combined with the earlier
E2E send fix (plain TEXT_MESSAGE_APP DM, firmware PKC) this closes archy↔archy LoRa.

**Open follow-ups:** #A surface received msgs under archy identity in all UI views; #6
device-onboarding modal; #8 Device-tab settings panel; #7 re-verify .116 in rotation;
#12 make modem_preset authoritative + hot-swap re-binding + RX-stall watchdog;
#14 signal-strength (RSSI/SNR) indicator per contact (from MeshPacket rx_rssi/rx_snr);
#15 map view plotting peer locations where shared (Meshtastic POSITION_APP portnum=3
lat/lon). See the resume memory `project_session_resume_2026_06_30_lora.md` for the full
task list.

### (historical) earlier TL;DR — RF-layer suspicion, now RESOLVED by the reboot-recovery fix
The **archy software is correct and deployed.** The blocker was at the
**radio/RF layer: the three radios are not hearing each other over the air at all.** No
amount of archy code change will fix that until the radios actually RF-link. **Resume by
testing the radios directly at home (Meshtastic phone app over Bluetooth) — see "DO THIS
FIRST AT HOME" below.** ← this turned out to be the want_config resubscribe bug above.

### What is DONE and deployed (commit pending — see below)
- **E2E send fix** (`core/archipelago/src/mesh/mod.rs` `send_message`, ~L1542): archy↔archy
  plain chat text is now sent as a **native `TEXT_MESSAGE_APP` DM** (firmware PKC-encrypts
  it E2E), NOT wrapped in our binary typed envelope. Archy peers' Sent rows are marked
  `encrypted=true` so the pill shows. Rich typed msgs still use `send_typed_wire`. This was
  the original root-cause fix (envelope-wrapped text silently broke archy↔archy LoRa).
- **NEW: software radio-reboot** end-to-end, so a wedged/RX-deaf radio can be rebooted
  without physical access (and for the Device-tab settings panel the user requested):
  - `meshtastic.rs`: `reboot(seconds)` driver method + `ADMIN_REBOOT_SECONDS_FIELD = 97`
    (verified vs meshtastic/protobufs admin.proto — `set_owner=32/set_channel=33/set_config=34`
    matched our existing constants, confirming the proto read).
  - `listener/mod.rs`: `MeshCommand::RebootRadio { seconds }`.
  - `listener/session.rs`: device-enum `reboot()` dispatch (Meshtastic only) + handler arm.
  - `mesh/mod.rs`: `MeshService::reboot_radio(seconds)`.
  - `api/rpc/mesh/messaging.rs`: `handle_mesh_reboot_radio` → RPC **`mesh.reboot-radio`**
    `{seconds?}` (default 2); dispatcher arm in `api/rpc/dispatcher.rs`.
  - `cargo check` passes. Built release **sha `ba4aed590027690d`** and DEPLOYED + active on
    `.116/.198/.228`. The RPC works (`{"reboot":true,"seconds":2}`).
  - ⚠️ **Caveat:** when called, archy logged "Sent Meshtastic radio reboot" but the radio did
    **not** visibly reboot afterward (no config re-stream). Either field 97 is still off, or
    newer firmware requires an admin session passkey even over local serial, or the USB serial
    stayed open through the 2s reboot so no reconnect was logged. **Needs on-device verification.**

### The hard evidence (why "nothing works")
- Directed DM tests `.198→.228` AND `.116→.228` (neither path reflashed): sender logs
  `Sent plain native DM dest=30d258436d65 part=1 total=1` and RPC returns `sent:true,
  encrypted:true`, but `.228` logs **nothing** — packet never reaches archy from the radio.
- A raw broadcast from `.198` (`mesh.broadcast`) was accepted by its radio but **not heard**
  by `.228`/`.116`.
- In an 8-minute window, **all three nodes received 0 inbound OTA packets from any other node.**
  Each only logs its OWN once-a-minute `Broadcast Meshtastic NodeInfo advert` + local TX
  `field=11` queue-status. `.228 mesh.status` = `messages_received:1` total.
- `.198`'s radio is alive and transmitting NodeInfo every 60s — so it's not dead; it's that
  **reception is broken on the receivers.** A radio cannot drop a broadcast AND a unicast to
  its own node number while config matches, unless it simply isn't on the same airwaves.
- archy provisioning is correct & identical across nodes (read back from device): PRIMARY =
  public LongFast (`name="" psk_len=1`), SECONDARY = `archipelago`, region=3 (EU_868). Admin
  field constants verified. The send path hands the radio a correct unicast MeshPacket
  (`to`=node, want_ack, hop_limit=3, plaintext `decoded` for the firmware to PKC-encrypt).

### PRIME SUSPECT (software-fixable) — modem-preset / frequency mismatch
archy only ever writes `region` + `use_preset` and **never explicitly pins `modem_preset`**
(it parses region but not preset; `set_lora_region` relies on the LongFast default). If ANY
radio has a non-default modem preset / frequency slot persisted (e.g. set via the Meshtastic
app, or a different factory default after the `.198` reflash), the radios are on **different
airwaves despite identical channel name + region**, and archy would never correct it.

### DO THIS FIRST AT HOME (decisive, ~2 min, only the user can do it)
Open the **Meshtastic phone app over Bluetooth** (works alongside archy's USB serial) on each
of `.116/.198/.228` and check:
1. Do the 3 nodes **see each other** in the node list (recent "heard")? → if NO, they're not
   RF-reaching (preset/freq/antenna/range).
2. Do all 3 show the **same** Modem preset (LongFast), Region (EU_868), Frequency slot, and
   the same PRIMARY channel? → any difference = the cause.
This single test separates "archy misconfigures the radios" from "radios physically can't
reach each other."

### THEN — the archy fix to apply (if preset/config differs)
Make archy **authoritatively write the full LoRaConfig** and force re-provision so all radios
converge: in `core/archipelago/src/mesh/meshtastic.rs::set_lora_region` (and its
caller/guard `ensure_lora_region` ~L304), explicitly set `modem_preset = LONG_FAST (0)` as a
field in the LoRaConfig (it's currently omitted/defaulted), and make the startup provision
path rewrite LoRa config when the preset doesn't match, then reboot the radio (use the new
`mesh.reboot-radio`). Also verify the `mesh.reboot-radio` actually reboots the radio
on-device (the caveat above).

### TEST RECIPE (works on each node)
- RPC helper used this session: a node-side `rpc.sh` that logs in (password
  `ThisIsWeb54321@`), grabs the `csrf_token` cookie, echoes it as `X-CSRF-Token`, and POSTs to
  `http://127.0.0.1:5678/rpc/v1`. Recreate it or run archy's RPC directly. Methods:
  `mesh.peers`, `mesh.status`, `mesh.messages`, `mesh.send {contact_id,message}`,
  `mesh.broadcast`, `mesh.reboot-radio {seconds}`.
- **LoRa contact ids:** `.116=1135977788` (prefix `3ca5b543`), `.198=3677050140` (`db2b551c`),
  `.228=1129894448` (prefix `30d25843`), stock `3ccc=1128152268`.
- **Link health check (run on each node):** look for inbound `from=Some("!...")` lines in
  `journalctl -u archipelago` that are NOT the node's own `Broadcast ... NodeInfo advert`. If
  zero across all nodes → RF link is down (the current state).
- **E2E success criteria:** send `.198→.228`, the marker appears in `.228` `mesh.messages` as
  an inbound row with `encrypted:true` / `transport:"lora"`, AND `.116↔.228` likewise.

### DEPLOY / BUILD RECIPE
- Build: from `core/`, `CARGO_TARGET_DIR=/tmp/archy-hotfix-target CARGO_INCREMENTAL=0 cargo
  build --release -p archipelago --bin archipelago`. (If `rust-lld: undefined hidden symbol`,
  it's incremental cache — `CARGO_INCREMENTAL=0` fixes it.)
- SSH key `~/.ssh/archipelago-deploy` is authorized on `.116/.198/.228`. SSH/UI/RPC password
  `ThisIsWeb54321@`. Per node: scp the binary, `sudo systemctl stop archipelago` →
  `kill -9 $(pgrep -x archipelago)` → `install -m0755` to `/usr/local/bin/archipelago` →
  `systemctl start archipelago`. Verify by `sha256sum` match + `systemctl is-active`.
- **Current deployed sha on all 3 = `ba4aed590027690d`** (the reboot-enabled build).

### Fleet state (as of 2026-06-30 PM)
- All 3 nodes on binary `ba4aed59`, active. Off-grid mode currently OFF (`mesh_only:false`).
- `.198` radio was reflashed to factory `firmware-heltec-v3-2.7.26` (recovered from corrupt
  NVS); region EU_868 persists. Its archy identity is NOT re-bound on `.228` (`.228` shows
  `.198` as raw radio "Meshtastic 551c", `arch_pubkey_hex` absent) because `.228` hasn't heard
  `.198`'s identity broadcast — a downstream symptom of the dead RF link, not a separate bug.
- The radios are powered & each transmitting; they are simply not hearing each other.

### Deferred UI (after LoRa works)
- Device-tab **settings panel** (gear/desktop) — host the "Reboot radio" button there; calls
  `mesh.reboot-radio`. Scoping done: add to the Mesh.vue actions row (mirrors Broadcast/Off-Grid
  buttons) + a `rebootRadio()` method in `neode-ui/src/stores/mesh.ts`. See `Mesh.vue` ~L1484
  actions row and `mesh.ts` ~L373 `broadcastIdentity()` pattern.
- Device-onboarding modal (detect plugged-in radio).

---

Current scope:
- Preserve existing mesh work: E2E indicators, FIPS/Tor transport indicators, typed-message paths, Meshtastic region/channel provisioning, and dirty Meshtastic receive-attempt changes.
- Take over the `3ccc` stock Meshtastic peer bug: LoRa text from `3ccc` to Archipelago `.116` does not surface in `mesh.messages`.
- Keep release-gate fixes already made in this session.

Local gate status so far:
- `cargo test -p archipelago --bin archipelago`: green, 849/849 after Meshtastic fixes.
- `python3 scripts/check-app-catalog-drift.py --release --strict`: green.
- `npm run type-check`: green.

Key changes made so far:
- Added cascade uninstall progress truthfulness assertion to `tests/lifecycle/bats/cascade-uninstall.bats`.
- Fixed release catalog drift filters and regenerated catalog metadata.
- Fixed invalid `apps/fedimint-clientd/manifest.yml` `cpu_limit` schema value.
- Updated stale/tight Rust tests without changing production behavior.

Remaining non-automatable / operational gates:
- Workstream B signing is blocked on the offline `RELEASE_MASTER_MNEMONIC`; code + runbook exist, but the publisher must pin/sign the release-root catalog.
- Phase-3 Quadlet backend rollout is implemented behind `use_quadlet_backends` and default-off. The gate skip-passes until explicitly enabled on a node; flipping it fleet-wide requires a coordinated flag rollout plus backend reinstall/migration verification.
- `.116` read-only `use-quadlet-backends-install.bats`: 6/6 skip-clean; no backend `.container` units, so Phase-3 is not active on that node.
- Release metadata still says `1.7.99-alpha` in `releases/manifest.json`; changelog top is `v1.8.00-alpha`. Cutting an actual 1.8.0 OTA requires an explicit version/manifest update.

Do not discard:
- `core/archipelago/src/mesh/listener/decode.rs`
- `core/archipelago/src/mesh/listener/session.rs`
- `core/archipelago/src/mesh/meshtastic.rs`

3ccc bug current hypothesis:
- The prior attempted Meshtastic fix added a hard stale-packet filter using `rx_time`.
- Stock Meshtastic radios without GPS/RTC can report tiny nonzero epoch values until time sync.
- That would make live `3ccc` packets look older than 10 minutes and get dropped before `mesh.messages`.
- Current patch treats implausibly early `rx_time` values as unknown rather than stale.

.116 live validation after 2026-06-30 hotfix:
- `.116` reachable by SSH; `archipelago` active; `/dev/mesh-radio -> ttyUSB0` attached.
- Current canary deploy is commit `b4531bb4`; backend sha
  `4ab53e539d89679ef664401a9a57996267772fed02327abc2912c3e77543acbf`; frontend bundle
  `index-YOAeJF7w.js` / `Mesh-BSAo88jN.js`.
- `main` pushed to `gitea-vps2`.
- RPC on `.116`:
  - `transport.status` currently reports `mesh_only:false` (off-grid mode is not enabled unless
    the user toggles it).
  - `mesh.status` reports Meshtastic connected: `device_type:"meshtastic"`,
    `self_node_id:1135977788`, `peer_count:13`.
  - Recent `.116` -> `3ccc` sent rows are stored with real 2026 timestamps and `transport:"lora"`.
- UI/backend fixes included in `b4531bb4`:
  - `transportLabel("lora")` displays **LoRa**.
  - mesh sends refetch messages after send so transport pills settle without browser refresh.
  - off-grid mode blocks the mesh-chat FIPS/Tor federation fallback and forces LoRa-only sends;
    banner text is `Tor/FIPS disabled - LoRa only`.
  - empty mesh-chat placeholder opacity reduced.
- Meshtastic diagnostics now identify the remaining blocker:
  - 3ccc NodeInfo is discovered:
    `Meshtastic peer is PKC-capable (NodeInfo public_key) node=1128152268 key_len=32`.
  - Bytes from stock Meshtastic text reach `.116`, but the custom parser rejects the packet:
    `Meshtastic FromRadio.packet did not parse into a decoded MeshPacket len=73 head=0dcc3c3e43153ca5b5432a16df56cbed`.
  - Non-text packets decode and are ignored with port numbers (`portnum=3/4/5`), so the serial
    read path is alive. Resume inside `core/archipelago/src/mesh/meshtastic.rs::parse_mesh_packet`.
- LoRa is therefore **not fully fixed** yet: stock `3ccc` -> `.116` text does not surface in
  `mesh.messages`, and `.116` -> `3ccc` still needs user-visible confirmation in the Meshtastic app.