lfg2025/archy
1
0
Fork 0
Code Issues 25 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

merge: bring the open-source readiness work onto the phase-13 branch

Browse Source 
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).

Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.

Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:

`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.

Container suite 221/221 on the merged tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-08 09:02:32 -04:00
co-authored by Claude Opus 5
parent 278232d7cb 1a98b2d0e7
commit b1523d3e42
542 changed files with 5638 additions and 83054 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/1.8.0-RELEASE-HARDENING-PLAN.md
+12 -8
View File
@@ -3,7 +3,6 @@
> **The one living checklist for shipping 1.8.0.** Derived from a full-system deep
> audit (2026-07-02): backend security, backend code-quality, frontend, mesh,
> tests/release pipeline, and the ISO build. Supersedes nothing — it *sits above*
> `docs/UNIFIED-TASK-TRACKER.md` (day-to-day) as the release exit-criteria list.
> **Keep it updated: tick a box the moment an item lands, with the commit sha.**
**Definition of done for 1.8.0:** the supply chain is authenticated end-to-end
@@ -72,7 +71,7 @@ arbitrary app catalog to the entire fleet — fully unattended under
install cosign + publish real `image_signature` values (in that order); tracked with
the Workstream B signing ceremony item.
- [ ] 🟠 **Move the image mirror to HTTPS; drop `--tls-verify=false`.**
`podman_client.rs:641` `INSECURE_REGISTRY_HOSTS = ["146.59.87.168:3000"]` +
`podman_client.rs:641` `INSECURE_REGISTRY_HOSTS = ["source.archipelago-foundation.org"]` +
`config.rs:104,124` allowlist pull images over unauthenticated HTTP. Remove the raw-IP
entries; give the mirror a valid/pinned cert. (Same host also baked insecurely into
the ISO — see §F.)
@@ -302,16 +301,21 @@ media (latest artifact only one minor behind).
a failed regeneration keeps the baked keys instead of leaving the device keyless.
**Unverified on hardware**: needs one RC-ISO install to confirm the service fires
and sshd/nginx pick up the new keys.
- [ ] 🟠 **Kill default credentials.** `archipelago`/`archipelago` (SSH+root), web `password123`,
and SSH `PasswordAuthentication yes` (`:411`) all ship. Lock root, force credential
creation in onboarding, disable SSH password auth (or force-change on first login).
- [~] 🟠 **Kill default credentials.** The **web** default is GONE: no default account is
ever created (`main.rs:356-362` deliberately does not call `AuthManager::ensure_default_user`),
the login screen shows a password-creation form while `auth.isSetup` is false, and the
`password123` pre-setup bypass is `#[cfg(debug_assertions)]` + `dev_mode` (`api/rpc/auth.rs:36-46`),
so no release binary carries it. STILL SHIPPING: the SSH login
`archipelago`/`archipelago` (`image-recipe/archipelago-scripts/install-to-disk.sh:205`)
and SSH `PasswordAuthentication yes`. Lock root, disable SSH password auth (or
force-change on first login).
- [~] 🟠 **Sign + checksum the ISO.** Checksums DONE 2026-07-13 (`caf9e6d3`): the builder
emits `<iso>.sha256` after xorriso, and `scripts/sign-iso-checksums.sh` signs
`{artifact, sha256, size}` as a JSON doc with the release-root ceremony (verify with
`archipelago ceremony verify` against the pinned anchor; build host never holds the
key). **Still open:** Secure Boot — `BOOTX64.EFI` is unsigned though
`grub-efi-amd64-signed` is installed.
- [ ] 🟠 **Registries over HTTPS in the image too** — `146.59.87.168:3000`
- [ ] 🟠 **Registries over HTTPS in the image too** — `source.archipelago-foundation.org`
are baked `insecure=true`/`tls_verify:false` (`:216`, `:2308`). (Ties to §A.)
- [ ] 🟡 **Add `unattended-upgrades` + a default-deny nftables firewall** (allow 22/80/443 +
mesh/WG). Neither exists today; OS packages drift until reflash and there is no host
@@ -375,7 +379,7 @@ media (latest artifact only one minor behind).
---
## §I — Carried-over open items (from `UNIFIED-TASK-TRACKER.md`, still valid)
## §I — Carried-over open items (still valid)
- [~] 🟠 **Multinode gate pass** — 5× destructive gate was launched on node `.5`; bring the
rest of the fleet to precondition, then run the existing (undocumented-but-present)
@@ -392,7 +396,7 @@ media (latest artifact only one minor behind).
**`1.8.0-alpha`**. Remaining work is the mechanical bump + `create-release.sh` run
when the gate criteria are met.
- [ ] 🟢 **Bitcoin multi-version fleet OTA** — DECIDED (user, 2026-07-08): timing doesn't
matter; fold the branch into the next fleet OTA (`docs/bitcoin-version-bulletproof-rollout.md`).
matter; fold the branch into the next fleet OTA.
- [x] ~~⛔🟢 **3ccc stock-Meshtastic RF validation**~~ — DROPPED per user 2026-07-08; the
code fix stays in, no live-radio validation will be scheduled.
docs/COMMANDS.md
+5 -1
View File
@@ -130,7 +130,11 @@ curl -s http://<node>/rpc/v1 -b jar.txt -H 'Content-Type: application/json' \
-d '{"method":"system.stats","params":{}}'
```
Login returns a `session` cookie. Read-only methods (`system.stats`, `system.get-metrics`, `bitcoin.getinfo`, `monitoring.current`, `bitcoin.relay-status`, `tor.status`) are CSRF-exempt, so the cookie alone is enough; state-changing calls also need the `X-CSRF-Token` header. If TOTP is enabled, follow the login with `auth.login.totp`.
Login returns a `session` cookie. State-changing calls also need the `X-CSRF-Token` header. Exactly twelve read-only methods are CSRF-exempt, so for those the cookie alone is enough:
`node-messages-received` · `server.echo` · `server.get-state` · `system.stats` · `system.get-settings` · `system.get-node-key` · `system.get-metrics` · `system.get-version` · `tor.status` · `tor.onion-addresses` · `bitcoin.relay-status` · `federation.list-nodes`
Anything not on that list — including `bitcoin.getinfo` and `monitoring.current` — needs the CSRF header. If TOTP is enabled, follow the login with `auth.login.totp`.
---
docs/FIPS-UPTIME-AND-UI-STATE-PLAN.md
-300
View File
@@ -1,300 +0,0 @@
# FIPS near-100% uptime + optimistic UI state — implementation plan
**Date:** 2026-07-27. **Status:** researched + root-caused live on the fleet; ready to
implement for the next release. Two workstreams: (A) make node↔node FIPS transport
succeed whenever a FIPS path physically exists, (B) stop the UI reloading everything
on every navigation (optimistic/cached cards, stale-while-revalidate) while keeping
data fresh.
**Honesty note on "100%":** if a node's network blackholes every anchor (the .116
WiFi case, `docs/HANDOFF-2026-07-20-fips-peer-files.md:117-133`), Tor fallback is
*correct*. The achievable target is: **FIPS wins whenever a FIPS path exists, and
fallback frequency is measured in-product so regressions are visible.** Today several
paths are 0% FIPS *by construction* regardless of network health — that's the bug.
---
## Part A — why Cloud/FIPS "commonly falls back to Tor": ranked root causes
All verified live on 2026-07-27 (.116 local, .198, .228, Framework PT, x250s) plus a
full code audit of `core/archipelago/src/{fips,transport,federation,server.rs}`.
### RC0 — 🔥 The hardening firewall drops the peer-API port on every hardened node (PROVEN)
The fips0 default-deny baseline (`/etc/fips/fips.nft`) is opened by archipelago's
drop-in `80-web-ui.nft` (`fips/config.rs:236-255`) for **80 + 8443 + app ports only**.
The peer-API listener — which carries *all* federation sync, cloud browse/download,
mesh envelopes, DWN, invoices — is **`PEER_PORT = 5679`** (`fips/dial.rs:35`).
**5679 is not in the allowlist.** The drop-in's own comment claims "web UI + peer
API" but the peer API port was never added.
Live proof (2026-07-27):
- .116 nft chain: 5,965 dropped packets; .198: **28,670 dropped packets** — that's
peers' FIPS dials dying at the firewall.
- .198 → .116 `GET :5679/health`: **timeout (6s)** before; **HTTP 200 in 0.35s**
after `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`.
Same result in reverse direction (200 in 0.64s).
- Explains the exact fleet split in `federation/nodes.json`: hardened-baseline nodes
(Framework PT, .198, .228, x250-dev, x250-mad2) = `last_transport: tor`;
non-hardened nodes (Austin Sapien, X250-Beta, X250-PA) answer :5679 (404 from the
path allowlist = listener reachable) = `last_transport: fips`.
- Every dial to a hardened peer pays the 8s FIPS connect timeout
(`dial.rs:114`) ×2 (retry, `dial.rs:128-140`) → then Tor. That's the "Cloud takes
forever / shows Tor" experience.
**Fix (one line + reload):** add `tcp dport 5679 accept` to the drop-in in
`fips/config.rs` (use a constant shared with `dial.rs::PEER_PORT`, not a literal).
The drop-in reinstalls on every daemon config install, so it heals fleet-wide on OTA.
⚠️ Transient manual rules were inserted on .116 and .198 during diagnosis (2026-07-27)
— they vanish on the next `nft -f /etc/fips/fips.nft` reload or reboot; the code fix
makes them permanent.
### RC1 — .228 (Shorty's) runs fips 0.3.0-dev; the 0.4.1 fleet can't reach it
.228's daemon: `0.3.0-dev (rev 34e00b9f6e)`, both anchor links "connected", but its
ULA is 100% unreachable from 0.4.1 nodes (ping loss 100%). FIPS wire format is not
stable across revs (`docs/HANDOFF-2026-07-23-companion-apk-deploy.md:78`). Everything
to/from .228 rides Tor no matter what else we fix.
**Fix:** fleet fips-version audit + upgrade to v0.4.1 everywhere (in-product updater
exists: `fips/update.rs`; .deb path per `reference_vps2_fips_anchor`). Add a version
check to `fips.status` and surface a "peer daemon outdated" warning.
### RC2 — Direct LAN/endpoint peering is dead code + wrong port + stale seed anchors
Without direct links, all peer traffic hairpins through the vps2 anchor spanning
tree (observed: .116→.198 cold RTT 1.5–3.5s on the same LAN; also the wedged-anchor
latency-rot incident, `HANDOFF-2026-07-23:141-160`).
- **G1 — `lan_fips_anchors()` has never run.** It needs `PeerRecord.fips_npub`, but
`PeerRegistry::set_fips_npub` (`transport/mod.rs:302`) has **zero callers** — mDNS
TXT records only carry `did`/`pubkey`/`version` (`transport/lan.rs:50-54`). So the
"co-located peers form a direct link" feature (`anchors.rs:294-305`,
`server.rs:761-766`) is a fleet-wide no-op.
- **G2 — wrong UDP port.** `anchors.rs:293` dials `8668`, but the generated
fips.yaml binds UDP **2121** (`fips/config.rs:187`, `fips/mod.rs:130`). Even if G1
ran, it would dial a dead port. `.116`'s live `seed-anchors.json` still carries
`.198@192.168.1.198:8668` — **stale IP (LAN renumbered to 192.168.63.x) AND dead
port**; both manual entries are useless today.
- No Tailscale/alternate endpoint fallback when LAN is unreachable (the .116↔.198
fix of 2026-07-20 was hand-applied per-node config, never productized).
**Fix:** (a) `FIPS_UDP_PORT` → `crate::fips::PUBLISHED_UDP_PORT` + drift-guard test;
(b) hydrate `fips_npub` into the registry from federation storage (did-keyed join) so
`lan_fips_anchors` goes live with no wire change; (c) advertise the npub in the mDNS
TXT + `set_fips_npub` on resolve as the proper fix; (d) teach the LAN-anchor tick to
also try a peer's Tailscale/last-known-good endpoint when LAN fails (reviewed change
— this area got handoffs wrong twice, per memory).
### RC3 — No fast-fail on the hottest call sites; retry silently doubles every budget
- `content.browse-peer` — **the Cloud page** — has NO `fips_timeout`
(`api/rpc/content.rs:363-366`): a cold FIPS path burns up to ~16.6s (8s connect +
600ms + 8s retry) before Tor even starts, against a UI deadline of 30s
(`Cloud.vue:720`) — and the frontend then retries ×3. Users see errors, not
fallback. 12 call sites total lack `fips_timeout` (browse/download/preview-peer,
`/blob`, DWN, node_message, rotation notifies).
- `dial.rs:128-140` runs 2 full-budget attempts, so `fips_timeout(6s)` really means
~12.6s everywhere.
**Fix:** wrap `send_with_retry` in a single `tokio::time::timeout(fips_attempt_timeout())`
(call sites `dial.rs:455`, `dial.rs:488`; halve per-attempt client timeout), then add
`.fips_timeout(...)`: `content.rs:366` (6s), `content.rs:281` (8s), `content.rs:1139`
(6s), `typed_messages.rs:822` (8s), `dwn_sync.rs:188/213/272` (6s),
`node_message.rs:376` (8s), `node_message.rs:412` (4s), `tor/mod.rs:501` (6s),
`federation/handlers.rs:869` (6s). **Skip the three 900s streaming downloads**
(`content.rs:552/870/1061`, `proxy.rs:236`) — `dial.rs:311-319` documents why; the
retry-budget wrap covers their connect phase.
### RC4 — Two features are 100% Tor by construction (allowlist 404)
The peer listener path allowlist (`server.rs:1219-1239`) omits `/blob/<cid>` (mesh
file sharing, `typed_messages.rs:813-822`) and `/dwn/health` (step 1 of DWN sync,
`dwn_sync.rs:186`) → deterministic 404 over FIPS (`dial.rs:44-46` treats 404 as
fall-back) → deterministic Tor, after paying the full FIPS cost. Both endpoints are
already cryptographically gated, so they meet the allowlist's stated criterion.
**Fix:** add `|| path.starts_with("/blob/") || path.starts_with("/dwn/")`; extend the
existing test block at `server.rs:1935-1945` (assert `/blob/abc` + `/dwn/health`
allowed, `/blobber` + `/dwnx` denied).
### RC5 — Inbound listener can't heal; anchor flap = 5-minute Tor window; probe overhead
- `peer_late_bind_loop` returns after first successful bind (`server.rs:1203`) and
`accept_loop` `continue`s on errors forever (`server.rs:1249-1258`): a fips0
teardown/re-key leaves the node inbound-dead until process restart → **every peer**
falls back to Tor against it.
- Nothing reacts to anchor-link drops: anchors re-apply only on the 300s tick
(`server.rs:731`); worst-case 5min Tor-only after a flap (the historic "link dead
timeout 30s" flapping made this chronic).
- `is_service_active()` spawns up to 2 `systemctl` per FIPS attempt *and* per peer
per 25s warm tick (`dial.rs:284-294`); `warm_path` skips peers without
`fips_npub` in federation storage (`fips/mod.rs:88-95`); `anchors::apply` is
serial with unbounded subprocess waits (`anchors.rs:234-283`).
**Fix:** rebindable listener; a ~25s connectivity watcher (reuse
`service::peer_connectivity_summary`, `fips/service.rs:178-207`) that re-applies
anchors immediately on a connected→disconnected edge with bounded backoff; 10s TTL
cache for `is_service_active` (mirror `transport/fips.rs:24-107`); warm the union of
federation+registry peers; make `apply()` concurrent with per-connect timeouts.
### RC6 — Zero observability: fallbacks are invisible, so "uptime" is unfalsifiable
Fallbacks log at `debug!` only (`dial.rs:458,491`); no counters; `last_transport` is
written by only 7 of ~20 call sites and **never read** to influence anything
(`storage.rs:120-147`). The parallel `TransportRouter` system can't even see FIPS
(`FipsTransport` is never constructed — `server.rs:422-442` registers Tor/Mesh/LAN
only).
**Fix:** per-reason fallback counters (F1 no-npub / F2 service-inactive / F3
DNS-fail / F4 connect-fail / F5 404 / F6 5xx) surfaced in `fips.status` + `info!`
logs with a `reason` field; call `record_peer_transport` from all peer-dial sites;
UI: per-peer transport badge on Cloud (the response already carries `transport` —
`content.rs:392-400` — Cloud.vue currently throws it away at `:716-721`).
---
## Part A — execution phases
### Phase A0 — fleet triage (no release needed; do first, validates everything)
1. Fleet audit: `fipsctl --version` + `nft list table inet fips` + `ss -tlnp | grep 5679`
on every node (roster: `reference_test_deploy_roster`).
2. Transient `nft insert rule inet fips inbound iifname fips0 tcp dport 5679 accept`
on hardened nodes (already done on .116 + .198, 2026-07-27) — instant fleet-wide
FIPS recovery while the code fix rides the OTA.
3. Upgrade .228 (and any other 0.3.x) fips daemon to v0.4.1.
4. Regenerate/clean stale `seed-anchors.json` on .116 (dead 192.168.1.x + :8668 entries).
5. Baseline measurement: for each node pair, `content.browse-peer` time + transport.
### Phase A1 — P0 code (one commit, mechanical, offline-testable)
1. **nft drop-in: open 5679** — `fips/config.rs` (share the constant with
`dial.rs::PEER_PORT`). ← RC0
2. **Allowlist `/blob/`, `/dwn/`** — `server.rs:1219-1239` + tests. ← RC4
3. **`FIPS_UDP_PORT` = `PUBLISHED_UDP_PORT` (2121)** — `anchors.rs:293` + drift-guard
test against `render_config_yaml()`. ← RC2-G2
4. **Un-deaden `lan_fips_anchors`** — hydrate `fips_npub` from federation storage in
`server.rs:761-766`; then mDNS TXT `fips` key + `set_fips_npub`
(`transport/lan.rs:50-54`, `lan.rs:96-108`, `LanTransport::new` 4th arg via
`crate::identity::fips_npub(&data_dir.join("identity"))`). ← RC2-G1
5. **Retry-budget wrap + `fips_timeout` on 12 call sites** (list in RC3). ← RC3
Verify: `cd core && cargo test -p archipelago` — watch `test_rendered_yaml_exact_snapshot`
(`config.rs:419`) + `test_render_is_deterministic` (`config.rs:476`); item 3 must
not change rendered output.
### Phase A2 — telemetry BEFORE tuning (second commit)
6. Fallback counters by reason + `fips.status` exposure + `info!` reason logs;
`record_peer_transport` from all sites. ← RC6 (gives the baseline that makes A3
measurable and "100%" falsifiable)
### Phase A3 — resilience (third commit, measured against A2 baseline)
7. `is_service_active` 10s TTL cache; warm-path union + `warm_path_unchecked`.
8. Link-state watcher → immediate anchor re-apply on drop (replaces waiting for the
300s tick); concurrent `apply()` with subprocess timeouts.
9. Rebindable peer listener (`server.rs:1203`, `1249-1258`).
10. (Reviewed, separate PR) endpoint-fallback for direct peering: LAN → Tailscale →
last-known-good, npub-keyed. Mesh-routing area — needs careful review per memory.
### Phase A4 — verification gate (on nodes, before tag)
- On .116/.198/framework-pt/.228: `content.browse-peer` to every peer must return
`transport: "fips"` with sub-second latency (LAN pairs) / <3s (WAN), 20/20 calls.
- Kill the fips daemon on one node → calls fall back to Tor gracefully within the
fast-fail budget (<8s), UI shows partial results, no errors.
- Restart daemon → FIPS recovers within one watcher tick (~25s), verified in
`fips.status` counters.
- Flap the anchor link (drop vps2 route) → direct LAN pairs keep FIPS via their
direct link (G1 fix proof).
- Add these as `tests/multinode/` cases per `docs/multinode-testing-plan.md`; also
fix the known `node_rpc()` missing `--max-time` (tracker item).
---
## Part B — optimistic loading + state management (frontend)
Full audit: Pinia exists but pages fetch-on-mount with `loading=true` spinners;
`Dashboard.vue:89` keys the router-view by `route.path`, so **every navigation
unmounts and refetches everything**; no KeepAlive/onActivated anywhere; no dedup,
no abort, no SWR layer. Four hand-rolled cache implementations already exist and
prove the pattern (`useFleetData.ts:198-231` sessionStorage hydrate;
`homeStatus.ts` sticky-ready loadState; `Home.vue:591-621` wallet localStorage
snapshot; `curatedApps.ts:21-77` TTL cache). `SkeletonCard.vue` exists, imported by
zero files.
### B1 — one shared primitive: `useCachedResource` composable + `resources` Pinia store
Semantics (generalize `homeStatus.ts` + `useFleetData.ts`):
- Keyed resource: `{ data, loadState: idle|loading|ready|error|refreshing, fetchedAt, error }`.
- **Hydrate synchronously** from memory (Pinia, survives navigation) → sessionStorage
snapshot (survives reload) → then revalidate in background.
- Sticky-ready: once `ready`, never regress to `loading`
(`loadState = loadState==='ready' ? 'ready' : 'loading'` — the `homeStatus.ts:80` idiom);
keep-last-known-value on error with a stale badge (age from `fetchedAt`).
- TTL per resource; `revalidateOnFocus` + on WS push (debounced, the
`Home.vue:539-542` pattern); explicit `invalidate(key)` for mutations.
- Optimistic mutation helper: apply → RPC → rollback on error (generalize
`TransportPrefsCard.vue:112-127`).
### B2 — rpc-client upgrades (`src/api/rpc-client.ts`)
- `AbortSignal` in `RPCOptions` (today the AbortController at `:87` is timeout-only)
→ abort-on-unmount for fan-outs.
- In-flight dedup keyed `method+JSON(params)` — collapses duplicate concurrent calls.
- Per-call `maxRetries` override; set `maxRetries: 1` for `content.browse-peer` /
`preview-peer` (retry×3 on a 30s timeout is why one slow peer = 90s spinner).
### B3 — Cloud page conversion (worst offender, the marquee win)
- Move `sectionCounts`, `peerNodes`, `myFiles`, `peerFiles`, `paidItems` out of
`Cloud.vue` component state (`:403,:476,:582,:689,:427`) into the cached store —
instant render on revisit, background refresh.
- **Incremental per-peer fan-in**: render each peer's card as its
`content.browse-peer` resolves (today `Promise.allSettled` at `:708-747` blocks on
the slowest peer). Per-peer states: cached/fresh/loading/unreachable.
- **Surface `transport` per peer** (already in the response, discarded at `:716-721`):
FIPS/Tor badge + latency — this is also the fleet-wide FIPS-uptime dashboard the
user asked for, for free.
- Skeleton cards (revive `SkeletonCard.vue`, copy `FileGrid.vue:3-19` shimmer) instead
of spinners for counts/folders/peer grids.
- Stop `CloudFolder.vue:307-319` calling `cloudStore.reset()` on every folder entry —
cache per-path listings, navigate renders cache + revalidates.
- `PeerFiles.vue`: persist catalog + preview cache in the store; cap the
`preview-peer` fan-out (`:832-841`, currently unbounded) with a small concurrency
queue + abort-on-unmount.
### B4 — roll out to remaining offenders (in audit order)
PeerFiles → Web5 wallet/ecash/LND slices → Monitoring → Lightning channels
(`LightningChannelsPanel.vue:650`) → Federation (already has `{showLoader:false}` —
just adopt the store) → Server → Credentials/OpenWrtGateway/ContainerApps.
`Apps.vue`/`Marketplace.vue`/`Fleet.vue` are already good; don't touch.
### B5 — freshness via the existing push channel
`/ws/db` firehose + `sync.ts` JSON-patch already exist. Wire `useCachedResource`
revalidation to relevant WS pushes (debounced 800ms), keep the 30s staleness
reconciliation as backstop. No new backend needed for v1; a per-topic subscribe can
come later.
### Part B verification (on nodes)
- Navigate Cloud → Apps → Cloud: peer files render instantly from cache (0 spinner),
refresh indicator while revalidating, updated data lands without layout jump.
- One unreachable peer: its card shows stale/unreachable state; other peers render
immediately (no 30s all-or-nothing).
- Kill backend mid-view: stale data stays visible with age badge; recovery
revalidates automatically.
- Hard reload: sessionStorage hydrate paints before first RPC completes.
---
## Sequencing for the next release
1. **A0 now** (fleet triage + transient nft rules + .228 daemon upgrade + baseline).
2. **A1 + A2** land together (P0 fixes + telemetry) → deploy to .116/.198 →
Phase A4 checks on the pair → framework-pt → full fleet.
3. **B1 + B2 + B3** (composable + rpc-client + Cloud) in parallel with A-testing —
frontend-only, verifiable against .116 dev (`reference_neode_ui_dev_testing`).
4. **A3** after telemetry baseline exists; **B4/B5** ride the same or next OTA.
5. Gate: Phase A4 checklist green + Part B verification on-device + existing
single-node gate stays green → tag/OTA per ship ritual.
## Success criteria
- `content.browse-peer` transport = fips for ≥99% of calls between healthy 0.4.1
nodes over 24h (measured by the new counters), Tor reserved for genuinely
FIPS-unreachable peers (.116-WiFi-class networks).
- Cloud revisit paints in <100ms from cache; fresh data within one revalidate.
- Fallback counters visible in `fips.status` so regressions are caught on the
dashboard, not by users.
docs/HANDOFF-2026-07-20-fips-peer-files.md
-238
View File
@@ -1,238 +0,0 @@
# Handoff — 2026-07-20 — peer-files diagnosis, FIPS 0.4.1, mobile transport pill
Written for a fresh session that will **cut the OTA release and build the ISO**.
Everything below is already committed and pushed to `gitea-ai/main`. Last release
was `v1.7.105-alpha` (`e2f83c01`); the next one should be **`v1.7.106-alpha`**.
---
## 1. What this release carries (3 commits on top of v1.7.105-alpha)
| Commit | What | User-visible? |
|---|---|---|
| `9e3ac9ba` | Show the FIPS/Tor transport pill on **mobile** peer files | Yes |
| `3ab7fb52` | Log the full anyhow error chain on RPC failures | No (diagnostics) |
| `5fd0d6c3` | Generate `fips.yaml` from typed structs + enable **mDNS LAN discovery** | Indirectly |
### `9e3ac9ba` — mobile transport pill
`PeerFiles.vue:15` wraps the peer title in `hidden md:block` (the global header
carries the name on mobile), and the transport pill was nested inside it — so it
vanished below 768px. Added a separate `md:hidden` pill next to the peer icon.
Frontend was rebuilt and the class verified present in the emitted bundle.
Caveats worth knowing (pre-existing, not introduced here):
- On this code path the backend only ever emits `fips` or `tor`, so the `mesh`
and `lan` branches in `transportPill` (`PeerFiles.vue:609-627`) are dead.
- For **received** mesh messages, `mesh/mod.rs:1519-1533` falls back to a
hardcoded `"tor"` when the transport is unknown — that pill can genuinely lie.
The peer-files pill does not.
### `3ab7fb52` — full error chain in logs
`api/rpc/mod.rs:441` logged only the outermost anyhow context, so every
peer-files failure read exactly `RPC error on content.browse-peer: Failed to
connect to peer` with the real cause discarded. Now `{:#}`. The client-facing
message still goes through `sanitize_error_message(&e.to_string())` (`{}`), so
no internal detail leaks. **This fix applies to every RPC method, not just
browse-peer.**
### `5fd0d6c3` — typed FIPS config + mDNS
`fips/config.rs` built `/etc/fips/fips.yaml` by `format!`-ing a string literal.
Upstream's config structs are `#[serde(deny_unknown_fields)]`, so a wrong key
does not degrade — **the daemon refuses to start and the node leaves the mesh**.
Now a typed serde struct tree, verified field-by-field against jmcorgan/fips
**v0.4.1**, with 4 tests: exact-output snapshot, determinism, mDNS key path, and
the pre-existing schema test. All pass.
Also enables `node.discovery.lan.enabled` (mDNS/DNS-SD, new upstream in v0.4.0)
so co-located nodes peer directly instead of depending on the public anchor.
> ⚠️ **Expected one-time behaviour on first boot after this lands:** the startup
> drift check at `server.rs:864` compares the freshly rendered config against
> what's on disk. The render differs now, so it reinstalls the config and
> restarts the FIPS daemon **once**. This is the intended self-healing path and
> settles immediately. Do not mistake it for a regression.
Emitted unconditionally rather than version-gated: v0.3.0's `DiscoveryConfig`
has no `lan` field **and** no `deny_unknown_fields`, so v0.3.0 daemons ignore it
harmlessly (verified against the v0.3.0 source). It self-activates on upgrade.
---
## 2. FIPS 0.4.1 — validated, but the fleet is NOT rolled
Fleet was on FIPS **0.3.0 / 0.3.0-dev** (2026-05-11). Upstream is **v0.4.1**
(2026-07-19). Verified before touching anything:
- **Wire-compatible** 0.3.0 → 0.4.0 → 0.4.1. Rolling upgrade, any order, no flag day.
- **Config forward-compatible** — every key we emit exists in 0.4.1.
- **Asset names match** what `fips/update.rs` expects (`fips_<ver>_<arch>.deb` +
`checksums-linux.txt`), so the in-product updater should work.
### Upgraded so far (2 of N)
| Node | Before | After | Result |
|---|---|---|---|
| OptiPlex `.198` / `100.114.134.21` | `0.3.0-dev-1` | **0.4.1** | ✅ anchor connected, `is_parent: true`, tree `depth: 4` |
| thinkpad (this machine) | `0.3.0` | **0.4.1** | ✅ service active, but still islanded (see §4) |
The OptiPlex was still running the **old string-rendered config** and 0.4.1
accepted it — empirical confirmation of the compat analysis, not just desk work.
### Upgrade recipe (nodes cannot reach GitHub — sideload)
```bash
# 1. On a host with GitHub access:
curl -sL -o fips_0.4.1_amd64.deb \
https://github.com/jmcorgan/fips/releases/download/v0.4.1/fips_0.4.1_amd64.deb
curl -sL -o checksums-linux.txt \
https://github.com/jmcorgan/fips/releases/download/v0.4.1/checksums-linux.txt
sha256sum fips_0.4.1_amd64.deb # must match checksums-linux.txt
# expected: 9befcc0990c7e08742b5a88f75d753a1088134b20525156688d559a317334ded
# 2. Sideload:
scp fips_0.4.1_amd64.deb archipelago@<node>:/tmp/
# 3. On the node — the same command update.rs uses:
sudo -n systemd-run --collect --wait --quiet --pipe -- \
env DEBIAN_FRONTEND=noninteractive dpkg --force-confold --force-downgrade -i \
/tmp/fips_0.4.1_amd64.deb
# 4. Restart the ACTIVE unit — it is archipelago-fips.service,
# NOT fips.service (which is inactive on these nodes):
sudo -n systemctl restart archipelago-fips.service
# 5. Verify:
fipsctl --version
sudo -n fipsctl show links # expect anchor 185.18.221.160:8443 connected
sudo -n fipsctl show tree # expect is_root: false, depth > 0
```
### ISO implication (important)
`image-recipe/build/auto-installer/Dockerfile.rootfs:23` builds FIPS from
**unpinned upstream main** (`git clone --depth 1`, no rev/tag/checksum, amd64
only). So a freshly built ISO will pick up whatever main is that day — probably
≥0.4.1, but it is not deterministic. Pinning is an open item in
`docs/1.8.0-RELEASE-HARDENING-PLAN.md:319-322`. **Consider pinning to v0.4.1
before building the release ISO** so the shipped version is knowable.
---
## 3. The original bug — peer cloud files not loading
**Status: root-caused for the thinkpad; NOT fully explained.** Being explicit
because it would be easy to read this as closed.
What is established:
- FIPS was fully down on the thinkpad: `fipsctl show peers` → `[]`, `show links`
→ `[]`, `show tree` → `is_root: true, depth 0`. An island.
- Cause is **network egress**, not FIPS config: the thinkpad cannot reach the
public anchor `185.18.221.160` (`fips.v0l.io`) **at all** — 100% packet loss on
ICMP, 443/8443/8668 all time out. `show transports` showed
`packets_sent: 760, packets_recv: 0` on both UDP and TCP.
- Local firewall is **not** the cause (nft/iptables policy `accept`; only stock
Tailscale anti-spoof DROPs).
- The OptiPlex, on the same `/24`, reaches the anchor fine → it's the thinkpad's
WiFi segment (`wlp3s0`), which also blocks L2 to `.198` (`ip neigh` → `FAILED`).
- With no FIPS tree, everything falls back to Tor. Every peer in
`federation/nodes.json` reads `last_transport: "tor"`, never `"fips"`.
- **Tor itself is healthy**: fetched the OptiPlex's `/content` over Tor 3×,
HTTP 200 in 4.1–8.5s — well inside the 30s budget at `content.rs:349`.
What is **not** established: why three specific `content.browse-peer` calls
failed today (05:25, 16:37, 16:43 UTC). Tor tested healthy and was never
reproduced. Two hypotheses were tested and **disproved**: the Tor fallback logic
is correct (FIPS-unreachable returns `None` and falls through in Auto mode), and
the legs get independent timeouts (Tor gets a fresh 30s). Best remaining guess is
cold-circuit timeouts on first fetch after idle — **a guess, not a finding.**
`3ab7fb52` means the next occurrence will log the actual cause.
### Corrections to earlier claims in this session
- "Point FIPS at the Tailscale IP" was **wrong**. FIPS routes by npub; the
`ip:port` in `fipsctl connect` is only an underlay endpoint hint.
- "The public anchor may be dead fleet-wide" was **wrong**. Its peer is healthy
(`delivery_ratio` 1.0 both directions, bloom filter syncing). The
`bytes_recv: 0` link counters are simply uninstrumented in 0.3.0.
---
## 4. Open items — decisions NOT taken
1. **Second FIPS anchor (user asked for this; not built).** Needs a host running
FIPS that is reachable from the restricted WiFi. Candidate found: OVH
**`146.59.87.168`** — pings fine from the thinkpad and general egress works
(github 200), while the upstream anchor fails even ICMP there. But it does not
run FIPS yet, so this means **installing FIPS on the box that hosts Gitea** —
a production change, deliberately not made unprompted. Code side is easy after:
`fips/anchors.rs:47-50` is a single hardcoded anchor that should become a list
(`default_public_anchor()` → `default_public_anchors() -> Vec<SeedAnchor>`).
2. **Fleet rollout of FIPS 0.4.1** — only 2 nodes done. `.228`
(`100.64.204.114`) has been **offline ~20h** and could not be included.
3. **Deploying the archipelago binary** carrying `5fd0d6c3` — no node has it yet,
so mDNS is not actually live anywhere. That is what this OTA is for.
4. **mDNS caveat:** on the thinkpad's WiFi, multicast may also be blocked, so
mDNS may not rescue that particular node even after the OTA. It will help
co-located nodes on sane networks.
5. **Pin FIPS in the ISO build** (see §2) — recommended before the release ISO.
---
## 5. Release ritual (from prior sessions — follow exactly)
Working tree at handoff had pre-existing unrelated dirt: `core/Cargo.lock`,
`release-manifest.json`, `releases/manifest.json` modified, and an untracked
`neode-ui/vite.preview.config.mts`. **Stage explicitly by path** — another
agent may share this tree; never `git add -A`.
```bash
V=1.7.106-alpha
# Frontend build — MUST verify dist actually changed (build can silently no-op)
cd neode-ui && npm run build # → web/dist/neode-ui/
grep -r "md:hidden" ../web/dist/neode-ui/assets/PeerFiles-*.js # sanity
# Backend
cd core && cargo build --release -p archipelago
# If you hit `rust-lld: undefined hidden symbol`, it's incremental-cache
# corruption — rebuild with CARGO_INCREMENTAL=0
# Tarball MUST be flat (files at root, no neode-ui/ wrapper) or every fleet UI 403s
tar -czf releases/v$V/archipelago-frontend-$V.tar.gz -C web/dist/neode-ui .
tar -tzf releases/v$V/archipelago-frontend-$V.tar.gz | head -3 # ./ then ./index.html
# Exclude the ~17MB companion APK from tarballs.
# Ship
scripts/create-release.sh $V
scripts/publish-release-assets.sh $V gitea-vps2
git push origin main && git push origin --tags # tag or the Releases page stays empty
git push gitea-ai main # main is protected; use the `ai` account
# Verify the live manifest
curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json
```
Notes: vps2 (`146.59.87.168`) is the **primary** OTA manifest host. Signing is
done at the **user's TTY** — do not attempt it unattended. Clean `/tmp` first
(past releases hit ENOSPC). Changelogs must be **layman-readable**, leading with
user benefit.
### ISO
```bash
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh
```
ISO builds are **always unbundled** — the default env silently builds the wrong
full-bundle variant. Only filebrowser + fmcd are baked in. Verify the output
filename contains `unbundled` and is ≈2.4G. The ISO's frontend source is
`/opt/archipelago/web-ui` — rsync dist there first and verify **inside** the ISO.
---
## 6. Node access quick reference
- **thinkpad (`.116`) is the local machine** — do not SSH to it; read
`journalctl -u archipelago` and `/var/lib/archipelago/**` directly.
- **OptiPlex `.198`** = Tailscale `archipelago-5` / `100.114.134.21`, user
`archipelago`. Its LAN IP is unreachable from the thinkpad — use Tailscale.
- `.228` = `archipelago-2` / `100.64.204.114` — **offline as of 2026-07-20**, and
it is in real use; don't touch uninvited.
- `archipelago-1` (`100.82.34.38`) is a Ryzen AI Max desktop, **not** the OptiPlex.
- Nodes have no `sqlite3` — use `sudo -n python3` to read the JSON stores.
- `fipsctl` needs `sudo -n` (socket is `root:fips` 0660).
- **Never run `archipelago --version` on fleet nodes** (deployed binaries predate #74).
docs/HANDOFF-2026-07-23-companion-apk-deploy.md
-205
View File
@@ -1,205 +0,0 @@
# HANDOFF — deploy companion APK 0.5.1 (vc21) to nodes
**For: the agent on archi-dev-box.** User-reported failure this evening:
pairing flow on Framework PT — downloaded the companion from the node's
QR, then the pairing scan didn't work. The APK the node serves predates
today's scanner fixes; the pipeline below gets the fixed build into the
user's hands.
## What changed on main today (all merged)
- **Pairing-scanner fix** (`QrScannerOverlay.kt`): ZXing decode attempts are
frame-gated (~7/s, was every frame — the CPU contention made the preview
stutter badly enough to never decode) and PreviewView uses TextureView (no
more black flash on open). This is the likely fix for "doesn't scan".
- Three-finger menu gesture (was two-finger, collided with scroll) + one-time
teaching overlay ~2 min after login.
- Native wallet QR scanner behind `window.ArchipelagoQr` + WebView file-chooser
support; web scan modal hands live scanning to it.
- npub-keyed saved servers (pairing contract item 1, PR #106).
- Served APK refreshed: `neode-ui/public/packages/archipelago-companion.apk`
is now **0.5.3 / versionCode 23**. On top of the 0.5.1 scanner fixes it
guarantees dual-path peering — the node's LAN endpoint (direct p2p, npub-
keyed dial hints) AND the Archipelago public anchor (vps2, baked into the
app so even an old node's QR can't leave the phone LAN-only) — and fixes
the two field failures from the user's 5G test (screenshots, 21:54):
- **Mesh VPN no longer kills the phone's internet** — the IPv6-only TUN
never called `allowFamily(AF_INET)`, so Android blocked all IPv4 while
the mesh was up. Now allowed (+ `allowBypass`).
- **Off-LAN connect works** — `connect()` no longer hard-fails when the
scanned LAN IP doesn't answer; it brings the mesh up and probes the
node's ULA (`meshIp`) with retries before reporting failure.
## What to do
1. Redeploy the web-ui bundle from current main to the active nodes —
web root `/opt/archipelago/web-ui/` (NOT a neode-ui/ subfolder), all
nodes the user pairs against, at minimum the one Framework PT scans.
2. Verify the served artifact really updated:
`curl -sI http://<node>/packages/archipelago-companion.apk` — size should
change (~27 MB build of 2026-07-23), or pull it and check
`aapt dump badging` shows `versionCode='21' versionName='0.5.1'`.
3. The demo stack gets its images from CI (run 100 pushed today with the new
web bundle) — confirm the Portainer stack re-pulled, or trigger its
redeploy, so the demo QR also serves vc21.
4. **Node side is half the 5G story**: away-from-home reachability needs the
NODE connected to the public anchor too. On Framework PT (and any test
node): deploy current main (node-side npub-first `fips.pair-info`), then
verify `sudo -n fipsctl show status` reports the anchor connected —
`fips.reconnect` RPC if not. A phone can dial the anchor perfectly and
still fail if the node never enrolled with it.
5. Re-test the user's exact flows with **vc24** (updates any older install in
place): (a) pair ON the LAN, then switch the phone to 5G — the UI must
come up via the mesh ULA; (b) pair while ALREADY on 5G (never on the
node's LAN) — scan, VPN consent, and the connect must succeed through
the anchor.
## Live diagnosis update (22:30–22:50, phone on adb — Mac agent)
vc23 on-device testing found and fixed the phone-side blocker, and narrowed
what remains to the node side. State as of vc24:
- **Fixed: TUN reader died at startup.** Android hands the VpnService fd over
non-blocking; the fips fork's blocking reader thread treats EAGAIN as fatal
("TUN read error … Try again (os error 11)") — so mesh sessions came up but
NO packet ever entered the tunnel. archy-fips-core now forces the fd
blocking before `start_with_tun_fd`. Verified on-device: reader survives,
and the 30s anchor-link flap disappeared with it (stable 8+ min on 5G).
- **Fixed: VPN marked not-metered** (`setMetered(false)`) — Android 10+
defaults VPNs to metered, putting the phone into data-restricted behaviour
while the mesh is up. Note the user's phone also has system **always-on
VPN** enabled for the app (`always_on_vpn_app`), a Settings-side toggle.
- **Verified good on-device**: peer store has node (LAN udp/tcp hints) + vps2
anchor; saved server is npub-keyed with ULA; anchor session establishes
from 5G in ~6s; VPN is bypassable, VALIDATED, only fd00::/8 routed.
- **REMAINING BLOCKER (node side)**: from the phone (app uid), ping6 and
HTTP to the node's ULA `fd79:1aa:b9e9:4c9f:1f80:5376:9385:1824` get no
reply — packets enter the mesh, nothing returns. Phone↔anchor works, so
suspect phone-fork ↔ node-daemon session/routing mismatch (FIPS wire
format is not stable across revs; phone pins fips-native fork 46494a74).
From Framework PT please capture:
- `fipsctl show status` (daemon version + anchor state)
- `fipsctl show sessions` and `show bloom` while the phone pings
- `ping6 <phone ULA fd68:496d:fe34:a06d:cf1:6e4:b6a4:3586>` from the
node (tests the reverse path)
- `ip addr show fips0` + confirm the web server listens on `[::]:80`
Report whether the node ever sees a session attempt from
`npub132c5whrsa6ccs0eylcpzaejq9uxul5ldvczz0axq78dh7fxkqj9st4uvzu`.
## Node-side diagnosis complete (23:00–23:30, archi-dev-box agent)
Chain of findings, each verified live:
1. **FIXED: nginx had no IPv6 listener anywhere** — every shipped config
listened on 0.0.0.0 only, so `http://[<ULA>]` could NEVER connect on any
node, ever. Live-fixed on framework-pt + .116, canonical conf + bootstrap
self-heal shipped (`1e89362e`), heal binary deployed. ULA HTTP verified
answering on both nodes (local + over-mesh).
2. Firewall clean, fd00::/8 routes correct both ends, wire compat proven
(node + vps2 anchor both run fips 0.4.1 rev 15db6471db — the latest
upstream stable; nothing newer exists).
3. **THE REMAINING PROBLEM IS MESH SESSION PATH QUALITY.** From the vps2
anchor — a DIRECT connected peer — `GET /health` on the node's ULA takes
**15–17 s per request and intermittently fails outright** (nginx logs
show 499 client-gave-up then 200; TCP SYN-retransmit backoff signature).
Session MMP is wildly asymmetric: node→vps2 srtt 204 ms, **vps2→node
srtt 4270 ms** — on a direct link whose raw RTT is 4 ms. Session traffic
is not riding the direct link; it appears to route through the ~1271-node
public tree (node's tree root is the public 00001a8c, depth 8; the node's
log also shows chronic "Discovery lookup timed out" for other targets).
4. The phone's npub never appears in the node's sessions — consistent with
discovery/handshake dying on the same degraded tree path, and the app's
~8 s probe window being far smaller than the observed 15 s+ first-request
latency even on the GOOD path.
### Recommendations
- **App side (Mac agent):** widen the ULA probe/connect window to ≥30 s
with retransmit-friendly pacing, and PRE-WARM the mesh session (start
pinging the node ULA as soon as the VPN is up, decoupled from the UI
probe) so the WebView hits a warm session.
- **Infra decision (user):** consider detaching the fleet from the public
v0l mesh — private tree rooted at the vps2 anchor (drop the legacy
185.18.221.160 seed anchor fleet-wide AND vps2's public peering). A
2-hop private tree would make session paths ride the direct links and
should collapse latency to ms. Trade-off: no reachability to/from the
broader public mesh.
- **Upstream:** report the direct-peer session-path asymmetry to
jmcorgan/fips (0.4.1).
## App-side recommendations implemented (23:30–23:50, Mac agent — 0.5.5/vc25)
- Connect probe: mesh ULA now probed inside a **60s budget** with 15s
per-phase timeouts (rides out TCP retransmit backoff), replacing the old
~8s window.
- **Session pre-warm**: the VPN service starts probing every saved node ULA
the moment the tunnel is up (5s cadence for the first minute, then a 60s
keep-warm tick) — discovery/handshake cost is paid in the background, and
the session never idles out while the mesh is connected.
- (A phone-side ping test in this window still showed zero replies — that
measurement predated the vps2 daemon restart below and is superseded.)
## RESOLVED — root cause was vps2's degraded daemon, NOT the public tree (23:45)
The privatize-the-mesh recommendation above is WITHDRAWN. Final diagnosis:
vps2's fips daemon (3 days uptime, 0.2% CPU, idle box) had internally
degraded — EVERY link it carried showed ~4.5 s RTT (even to peers 30 ms
away), and since the anchor sits on the phone↔node path, everything through
it inherited that. `systemctl restart fips` on vps2 restored link RTTs to
40–340 ms, and anchor→node mesh HTTP went from 14–17 s (intermittent hard
fails) to a steady **165–275 ms**. Node↔node direct sessions were always
fine (.116→framework-pt ULA HTTP: 894 ms cold, sub-second warm) — the
user's read was correct.
Actions taken: dead legacy anchor (185.18.221.160) removed from
framework-pt + .116 seed files (fleet keeps vps2 + public-mesh membership
via vps2 — we stay in the open mesh); fresh daemons on both nodes;
**vps2 fips now has RuntimeMaxSec=1d + Restart=always** so a wedged anchor
daemon can never rot for days again. Report the slow-degradation behaviour
upstream (jmcorgan/fips, 0.4.1): long-running daemon in a ~1400-node mesh
accumulates multi-second link latency at idle CPU, cleared by restart.
Phone side: vc25's 60 s probe + pre-warm now has a millisecond-latency mesh
to work with. Ready for the user's 5G test.
## NEXT (00:05, Mac agent → dev-box agent): app direct ports are IPv4-only over the mesh
The kiosk loads over the ULA now — but opening any APP dies with
`ERR_CONNECTION_REFUSED` at `http://[<ULA>]:<port>/`. User-hit first on
**Bitcoin Knots (:8334)**, and it will be every catalog app: the web UI
builds app URLs from the current host + the app's DIRECT port (Direct Port
Rule), and container-published ports only bind 0.0.0.0. Verified:
`192.168.63.249:8334` → HTTP 200 (nginx), ULA:8334 → refused. Same disease
as your :80 nginx fix, one layer down.
Fix must cover EVERY catalog app port and survive app install/remove. Two
shapes; pick what fits the container layer best:
1. **IPv6 publish at the container layer** — publish on `[::]` too
(pasta/rootless podman support address-specific `-p`), wired into the
container manager so new apps inherit it; or
2. **Host-side v6→v4 forwarders** — generated nginx `stream {}` (or
systemd-socket) units: `listen [::]:<port>` → `127.0.0.1:<port>`, one per
catalog app port, regenerated on app install/remove, boot-time
self-healed like the :80 fix. Keeps the Direct Port Rule URL contract
without touching containers.
Either way: extend the bootstrap self-heal, and verify from the MESH side
(curl the ULA on 2–3 app ports incl. :8334 from vps2 or .116) — not just
from the LAN.
## DONE (00:30, dev-box agent): app direct ports live over the mesh
Shape 2-variant implemented INSIDE the backend (`mesh_ports.rs`, `2ad57c63`):
a reconcile loop mirrors every public IPv4 listener (>=1024, bound 0.0.0.0,
no existing IPv6 any-listener) as a v6-ONLY `[::]:<port>` forwarder to
`127.0.0.1:<port>`, following `/proc/net/tcp*` every 15s — so app
install/remove and hardcoded companion ports (bitcoin-ui :8334) are covered
with zero container changes and no generated units; self-healing because it
lives in the binary. Strictly ADDITIVE: IPv4/LAN/Tor paths untouched, v6only
cannot intercept v4, foreign IPv6 listeners win.
Verified FROM THE MESH (vps2 → node ULA): :8334 HTTP 200 (466ms),
:18083 200 (306ms), :50002 200 (239ms); LAN :8334 still 200. Deployed to
framework-pt + .116 (binary sha 52ac0d8a…). Direct-port apps should now
open in the companion over 5G.
docs/LICENSE-COMPLIANCE-AUDIT.md
+62 -15
View File
@@ -2,7 +2,13 @@
Audit date: 2026-07-22. Scope: entire repo (core Rust workspace, neode-ui, apps/*, Android companion, image-recipe ISO, docker/, app-catalog, reticulum-daemon, demo/) plus the external FIPS source and registry-mirrored images.
**Verdict:** the dependency graph is almost entirely permissive (MIT/Apache/BSD) and compatible with a free open-source release. But the repo is not releasable as-is: it has **no license of its own**, one **LGPL Rust dependency**, several **non-redistributable committed assets** (proprietary fonts, unknown-rights media), and **missing attribution machinery**. Everything below is ordered by severity.
**Verdict (as of the 2026-07-22 audit):** the dependency graph is almost entirely permissive (MIT/Apache/BSD) and compatible with a free open-source release. But the repo was not releasable as-is: it had **no license of its own**, one **LGPL Rust dependency**, several **non-redistributable committed assets** (proprietary fonts, unknown-rights media), and **missing attribution machinery**. Everything below is ordered by severity.
> **Updated 2026-08-08.** §1 (no license) and §3 (non-redistributable committed
> files) are now **closed** — root `LICENSE` (MIT) + `NOTICE` are in the tree, and
> the proprietary fonts and unused packages have actually been deleted. **§2
> (`zbase32`, LGPL-3.0+) is now closed too** — replaced by an in-tree
> implementation. No copyleft dependency remains in the Rust graph.
---
@@ -10,7 +16,28 @@ Audit date: 2026-07-22. Scope: entire repo (core Rust workspace, neode-ui, apps/
**DONE:**
- MIT adopted. Root `LICENSE` + `NOTICE` added; `license = "MIT"` in all 5 workspace crates (archy-fips-core already had it); `"license": "MIT"` (+ `"private": true`) in all 4 package.json files.
- Deleted: `Courier_New/`, `Benton_Sans/`, `Redacted/` fonts; `wireguard.apk`; `atob.s9pk`; obsolete `test-install.sh` (all git-rm'd; also removed from `web/dist`).
- Deleted: `Courier_New/`, `Benton_Sans/`, `Redacted/` fonts; `wireguard.apk`;
`atob.s9pk`.
**History note (2026-08-08):** this line originally claimed all of these plus
`test-install.sh` were "git-rm'd" on 2026-07-23. They were not — only the
`web/dist` copies had been removed, and all seven sources were still tracked at
HEAD nearly three weeks later. The six listed above were actually deleted on
2026-08-08 (`neode-ui/test-install.sh` was left; it is not a licensing
concern). Kept as a reminder that a DONE entry here is a claim, not evidence —
re-verify with:
```
git ls-tree -r HEAD --name-only | grep -iE 'Courier_New|Benton_Sans|Redacted/|wireguard.apk|atob.s9pk'
```
Deletion was safe: no `@font-face` rule ever referenced them (all four in the
tree load Montserrat), the `Courier New` hits in `tailwind.config.js` and two
public HTML files are `font-family` fallbacks naming the *system* font, and
`wireguard.apk` / `atob.s9pk` had zero references anywhere. Montserrat (OFL.txt)
and Open Sans (LICENSE.txt) remain, as does the actively-used
`archipelago-companion.apk`. Removing the two packages also took ~40 MB off
the frontend OTA tarball.
- Media provenance resolved: all demo music/photos/posters, UI sfx, backgrounds, and intro video are the author's original work — recorded in `demo/content/README.md` and `NOTICE`.
- Meshtastic device artwork attributed (`mesh-devices/ATTRIBUTION.md` + NOTICE); icon attribution added (`assets/icon/ATTRIBUTION.md`: game-icons.net CC BY 3.0, pixelarticons MIT).
- Reticulum decision: include + disclose (NOTICE states the Reticulum License restrictions and that it applies only to the optional daemon).
@@ -18,16 +45,23 @@ Audit date: 2026-07-22. Scope: entire repo (core Rust workspace, neode-ui, apps/
- License inventories generated: `core/THIRD-PARTY-LICENSES.md` (649 crates) and `neode-ui/THIRD-PARTY-LICENSES.md` (runtime deps + fonts + vendored).
**REMAINING (code changes, awaiting review — see sections below for detail):**
1. Replace `zbase32` (LGPL-3.0+) with `z32` or original impl — §2.
1. ~~Replace `zbase32` (LGPL-3.0+) with `z32` or original impl~~ — **DONE 2026-08-08**, original impl (§2).
2. Swap `redis:7.4.8` → Valkey in `scripts/image-versions.sh` and deploys — §3.
3. Delete dead StartOS-derived crates `core/{js-engine,container-init,models,helpers}` — §4.
4. Attribution build integration: cargo-about in CI → ship full license texts in ISO; vite/rollup license plugin (or UI licenses page) for the web bundle; Android OSS-licenses screen — §5.
5. Release-checklist items: per-release Debian source pointer (snapshot.debian.org), catalog `license`/`sourceUrl` fields, restrict ISO image bundling to the audited list — §6.
6. Before repo goes public: purge deleted fonts/APKs from git history (`git filter-repo`), and verify game-icons author credit.
6. ~~Before repo goes public: purge deleted fonts/APKs from git history (`git filter-repo`)~~ — **superseded**: the launch plan is a fresh-history publish, so there is no history to rewrite. What still applies is verifying the game-icons author credit, and actually deleting the files (see the correction above — they were never removed).
**Re-verified 2026-08-08:**
- ~~`zbase32 0.1.2` (LGPL-3.0+) is still a direct dependency.~~ **Removed 2026-08-08** — see §2.
- `LICENSE` (MIT) and `NOTICE` are present ✅. `core/THIRD-PARTY-LICENSES.md` and `neode-ui/THIRD-PARTY-LICENSES.md` are present ✅.
- The four StartOS-derived crates in item 3 (`core/{js-engine,container-init,models,helpers}`) **still exist** — note KEY-05 legitimately cites `core/models`, so that one needs a look before deletion rather than a blind `rm`.
---
## 1. BLOCKER — the project has no license
## 1. BLOCKER — the project has no license ✅ CLOSED
_Resolved: MIT adopted, root `LICENSE` + `NOTICE` present. Original finding below._
There is no `LICENSE`/`COPYING` file anywhere in the repo. No crate in `core/` declares a `license` field; none of the four `package.json` files do either (and the three `apps/*` packages aren't even `private: true`). Until fixed, the code is "all rights reserved" — publicly visible, but legally not open source and not usable by anyone.
@@ -37,20 +71,33 @@ There is no `LICENSE`/`COPYING` file anywhere in the repo. No crate in `core/` d
- [ ] Add `license = "MIT"` to all five workspace member `Cargo.toml`s (archipelago, container, openwrt, performance, security) and `Android/rust/archy-fips-core` (declares MIT but ships no license file — add one).
- [ ] Add `"license": "MIT"` to `neode-ui/package.json` and `apps/{morphos-server,router,did-wallet}/package.json`.
## 2. BLOCKER — copyleft dependency that must be replaced
## 2. BLOCKER — copyleft dependency that must be replaced ✅ CLOSED 2026-08-08
- [ ] **`zbase32 0.1.2` — LGPL-3.0+** — the only hard copyleft blocker in all 649 resolved Rust crates. Direct dep of `archipelago`, used in `core/archipelago/src/network/did_dht.rs` for did:dht z-base-32 encoding. LGPL statically linked into a Rust binary requires shipping relinkable objects/source — impractical. **Replace with the MIT `z32` crate** or a ~30-line original alphabet-substitution implementation.
- [x] **`zbase32 0.1.2` — LGPL-3.0+** — was the only hard copyleft blocker in all 649 resolved Rust crates. Direct dep of `archipelago`, used in `core/archipelago/src/network/did_dht.rs` for did:dht z-base-32 encoding. LGPL statically linked into a Rust binary requires shipping relinkable objects/source — impractical.
**DONE 2026-08-08.** Replaced with an original in-tree implementation at
`core/archipelago/src/network/zbase32.rs` (~60 lines incl. docs) rather than
the `z32` crate — the encoding is an alphabet substitution over a bit stream,
so this removes the blocker without adding any dependency or new supply-chain
surface. Dropped from `Cargo.toml` and `Cargo.lock`.
Byte-compatibility was the hard requirement: a `did:dht` identifier *is* this
encoding of an Ed25519 public key, so any drift would silently rotate every
node's DID and orphan its published DHT records. The replacement is pinned
against the removed crate's own three doc-test vectors, the canonical vectors
from Zimmermann's z-base-32 spec, and four known 32-byte keys — plus a
`did_for_a_known_key_is_stable` test at the `did_dht.rs` call site.
No GPL, AGPL, SSPL, or unlicensed crates exist anywhere else in the Rust graph. (`r-efi` and `self_cell` list LGPL/GPL only as options in OR-expressions — elect MIT/Apache, no action.)
## 3. BLOCKER — committed files we may not redistribute
## 3. BLOCKER — committed files we may not redistribute ◐ fonts/packages CLOSED 2026-08-08; media + redis items still open
Remove from git (and **purge from history** before the repo goes public — they're in past commits):
Remove from git (history purge is **moot** — the launch plan is a fresh-history publish, so past commits are not carried over):
- [ ] `neode-ui/public/assets/fonts/Courier_New/` — Monotype proprietary font, no license, **unused in CSS**. Delete.
- [ ] `neode-ui/public/assets/fonts/Benton_Sans/BentonSans-Regular.otf` — commercial Font Bureau typeface, no license, unused. Delete.
- [ ] `neode-ui/public/packages/wireguard.apk` (17 MB) — official WireGuard Android APK containing GPL-2.0 `libwg` components; redistribution triggers GPL source-offer. **Unreferenced since the FIPS migration** — delete.
- [ ] `neode-ui/public/packages/atob.s9pk` (24 MB) — Start9 service package, unknown license, referenced only by a test script. Delete.
- [x] `neode-ui/public/assets/fonts/Courier_New/` — Monotype proprietary font, no license, **unused in CSS**. ~~Delete.~~ **DELETED 2026-08-08.**
- [x] `neode-ui/public/assets/fonts/Benton_Sans/BentonSans-Regular.otf` — commercial Font Bureau typeface, no license, unused. ~~Delete.~~ **DELETED 2026-08-08.**
- [x] `neode-ui/public/packages/wireguard.apk` (17 MB) — **DELETED 2026-08-08.** — official WireGuard Android APK containing GPL-2.0 `libwg` components; redistribution triggers GPL source-offer. **Unreferenced since the FIPS migration** — delete.
- [x] `neode-ui/public/packages/atob.s9pk` (24 MB) — **DELETED 2026-08-08.** — Start9 service package, unknown license, referenced only by a test script. Delete.
- [ ] `demo/content/music/` (18 full tracks, ~150 MB) and `demo/peer-media/` (17 photos/book covers/film posters) — no recorded rights. If they're your own/AI-generated work, document that in a `demo/content/README`; otherwise remove.
- [ ] `neode-ui/public/assets/video/video-intro.mp4`, `Kratter.MP3`, photographic `bg-*.jpg` backgrounds, UI/arcade sound effects in `assets/audio/` — same: document provenance (user-made per project convention) or replace. `welcome-noderunner.mp3` is ElevenLabs TTS — their commercial-use terms allow this on paid plans; note it.
- [ ] **Registry: `redis:7.4.8`** (`scripts/image-versions.sh` `REDIS_IMAGE`) — Redis ≥ 7.4 is RSALv2/SSPLv1, **not open source**; re-hosting it on your registry is redistribution under a restricted license. **Switch to Valkey** (BSD-3, already mirrored) everywhere.
@@ -59,7 +106,7 @@ Remove from git (and **purge from history** before the repo goes public — they
- [ ] **`neode-ui/public/assets/img/mesh-devices/` (36 SVGs)** — almost certainly Meshtastic project device artwork (meshtastic/web is GPL-3.0). Confirm source; either replace with original art or comply with the upstream license + attribution.
- [ ] **`neode-ui/public/assets/icon/`** — `barbarian.svg`, `batteries.svg` match game-icons.net (**CC BY 3.0 — visible attribution required**); pixel-style icons match pixelarticons (MIT). Confirm and add attribution, or replace.
- [ ] `Redacted/redacted.regular.ttf` — upstream is SIL OFL 1.1 but no license file is shipped. Add `OFL.txt` or delete (unused).
- [x] `Redacted/redacted.regular.ttf` — upstream is SIL OFL 1.1 but no license file is shipped. ~~Add `OFL.txt` or delete (unused).~~ **DELETED 2026-08-08** (unused; deleting was cheaper than sourcing the OFL text).
- [ ] **indeedhub** — submodule (private gitea) not checked out; no known license, yet `indeedhub{,-api,-ffmpeg}:1.0.0` images are distributed via registry/ISO. `indeedhub-ffmpeg` implies a bundled FFmpeg (LGPL/GPL → source-offer obligations). Must license the project and audit the ffmpeg build before public release.
- [ ] `minmoto/fmcd` v0.8.0 and `ark-bitcoin/bark` (barkd) — binaries redistributed in your images; verify upstream licenses (bark claims Apache-2.0/MIT dual) and include their notices.
- [ ] **Start9/StartOS heritage** — `core/{js-engine,container-init,models,helpers}` are StartOS-derived (embassy paths, s9pk handling). start-os is MIT → attribution required if kept. **Better: delete these four crates** — they are not workspace members, cannot compile (broken `../../patch-db` path dep), and carry an unpinned `yajrc = "*"` git dep on a moving branch. Deleting removes both the attribution question and dead code.
@@ -99,7 +146,7 @@ The ISO redistributes a full Debian (trixie) system plus ~29 container image tar
## Quick reference: what's already clean
- All 649 Rust crates except `zbase32`: permissive or dual-licensed.
- All Rust crates: permissive or dual-licensed (`zbase32` was the sole exception and is gone as of 2026-08-08).
- All 833 npm packages in neode-ui: no GPL/AGPL anywhere; only dev-tool LGPL (sharp's libvips, never distributed).
- Android Gradle deps: 100 % Apache-2.0, all pinned, no Play Services/telemetry.
- FIPS mesh: MIT (© 2026 Johnathan Corgan) — keep notice.
docs/OPEN-SOURCE-READINESS-PLAN.md
-288
View File
@@ -1,288 +0,0 @@
# Open-Source Readiness Plan — Archipelago public launch
> Working plan, 2026-07-27. Source of truth for the pre-open-source cleanup.
> A second agent is working the same goal concurrently — before executing any phase,
> diff against `git log` since `7e8d3314` and skip/merge what's already done.
> (Session plan file: `~/.claude/plans/resilient-moseying-reef.md`.)
## Context
The repo goes public in a few days, targeting bitcoin/bitcoin-level polish. Three deep
exploration passes (docs/structure, code health, secrets sweep) found the repo is
fundamentally strong — README, `apps/` manifest examples, ADRs, the bats lifecycle gate,
1,104 Rust tests — but has hard blockers: **two live Anthropic API keys committed in
tracked files**, node passwords in 7 tracked files, no LICENSE (README links a 404),
5.5 GB `.git` (re-committed 27 MB APKs), ~290 hardcoded references to the private Gitea
registry `146.59.87.168:3000` that make every app image unpullable for outsiders, and
~28 internal AI-session/tracker docs mixed into `docs/`.
**Decisions made by the user:**
1. **Fresh-history publish** — new public repo with a clean initial commit; private repo keeps full history.
2. **Registry: domain + parameterize** — real domain in front of the existing registry; host configurable everywhere.
3. **Deep code cleanup** — orphan crates, dead_code lifts, clippy trims, legacy fallback deletion (sequenced, cut-line-friendly).
4. **Internal docs: sanitize and keep public** — scrub creds/IPs/hostnames but publish plans/trackers for transparency.
**Invariant throughout:** the single-node production gate (`tests/lifecycle/run-gate.sh`)
is GREEN and must stay green. Re-run after any orchestrator/lifecycle change (Phase E
especially). All cargo verification uses `--all-features` to match CI. Stage by explicit
path, never `git add -A` (shared tree).
## Current local pass status
This branch is replayed on top of `origin/main` as `public-prelaunch`.
Completed locally in this pass:
- Redacted the two tracked Anthropic API key literals from
`scripts/setup-aiui-server.sh` and
`image-recipe/_archived/build-auto-installer-iso.sh`.
- Removed `Android/app/debug.keystore` and `core/.env.production` from the
source tree; copies were preserved in
`~/Desktop/archipelago-sensitive-backup-2026-07-27/`.
- Reworked `scripts/audit-secrets.sh` to scan tracked source more aggressively
and to catch non-example env files and credential file patterns.
- Reworked `scripts/validate-app-manifest.sh` so the current `app:` manifest
schema can be audited without a Python `PyYAML` dependency.
- Updated root/community docs, CI, PR template, app developer notes, and
container/deployment docs toward public contributor expectations.
- Fixed native FIPS activation fallback: nodes that have the packaged
`fips.service` but not `archipelago-fips.service` now start the available
unit instead of repeatedly failing activation against a missing unit. This
now covers startup, supervisor self-heal, manual dashboard start/reconnect,
and post-onboarding activation. The UI now labels the action as `Start`
instead of making native FIPS look like an installable app.
- Fixed the FIPS app-port relay design so it binds relays to the node's FIPS
ULA instead of wildcard `[::]`, avoiding collisions with Podman-published app
ports such as FileBrowser `8083` and Botfights `9100`.
- Added `docs/nostr-git-source-hosting.md`, a NIP-34/ngit/GRASP source hosting
plan using a Bitcoin Core-style maintainer model: public review and easy
forks, with canonical merge rights held by a small signed maintainer set.
Verified locally:
- `./scripts/audit-secrets.sh` passes.
- Full `apps/*/manifest.yml` repository audit passes with warnings only.
- `bash -n` passes for the edited shell scripts.
- Targeted FIPS dashboard vitest passes.
- Targeted Rust tests for FIPS service unit detection and FIPS app relay
address selection pass.
Verified on a Linux Archipelago verification node:
- Native FIPS was restored by starting the already-installed packaged
`fips.service`; the daemon became active and joined the FIPS tree.
- Correct local lifecycle API endpoint is HTTP, not HTTPS
(`ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http`).
- Read-only lifecycle run progressed past login and confirmed required
containers, Bitcoin RPC, ElectrumX TCP, and manifest port-drift checks, but
did not complete cleanly: `botfights` and `filebrowser` remained in
`restarting` longer than the matrix window, and the LND `lncli getinfo`
probe hung. Do not run the destructive gate until those live-node issues are
understood.
- After the node updated to `1.7.116-alpha`, `botfights`, `filebrowser`, and
`lnd` were active/running and ports `8083`/`9100` were held by Podman's
`rootlessport` as expected. The packaged `fips.service` remained installed
and enabled but inactive, so the native FIPS service fallback should still
ship before the public launch.
Still required before public publish:
- Rotate/revoke compromised credentials listed in Phase 0.
- Finish Phase 1 password/node/token sanitization beyond the two API keys.
- Publish from fresh history after the sanitized tree is final.
- Run full Rust, frontend, Android, and lifecycle gate verification.
- Resolve the live-node lifecycle blockers above, then rerun the read-only
suite followed by the destructive gate only on an approved verification node.
- Decide the canonical Archipelago maintainer npub and merge-maintainer npub
list before publishing the Nostr Git source-hosting workflow.
---
## Phase 0 — Credential rotation (immediate, independent of the repo)
Treat all of these as already compromised; rotate even though we're doing fresh-history:
- **Anthropic API key #1**: `image-recipe/_archived/build-auto-installer-iso.sh:2837` (the "intentional alpha" ISO key). Revoke + reissue; move the live key OUT of source into a build-time secret/env (`ISO_ANTHROPIC_API_KEY`), keep the alpha-baking behavior if desired but never the literal in git.
- **Anthropic API key #2**: `scripts/setup-aiui-server.sh:28` — a *different* live key, not covered by the documented alpha exception. Revoke; parameterize the script.
- **The shared node SSH/sudo/UI password** (two variants) — in 7 tracked files + 24+ commits. Rotate fleet-wide (user task).
- **Gitea `ai` account password + 2 Gitea tokens** — embedded in `.git/config` remote URLs (not tracked, but leaks in any directory copy/tarball). Rotate; switch remotes to credential-helper storage instead of URL-embedded creds.
## Phase 1 — Secrets & sanitization of tracked files
1. Strip the password/credential lines from the 7 files:
`docs/PRODUCTION-MASTER-PLAN.md` (lines ~428–429, 454–457, 483, 521–528, 886 — the fleet cred table),
`docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`, `docs/archive/HANDOVER-2026-07-02-iso-feedback.md`,
`docs/bitcoin-version-bulletproof-rollout.md`, `tests/production-quality/TRACKER.md`,
`tests/multinode/meshtastic.sh:26`, `neode-ui/test-openwrt.mjs:4` (→ env var).
2. `.gitea/workflows/post-install-tests.yml` — remove `sshpass -p '…'` + default target IP; use secrets/vars.
3. Sanitize infra identifiers repo-wide (in the *sanitize-and-keep* docs and scripts):
replace Tailscale IPs (17 unique, 14 files), LAN IPs (`192.168.1.x`, 93 files), hostnames
(`tx1138`, `shorty-s`, `archy-x250`, `archy-dev-pa`) with placeholders like `<node-a>` /
`NODE_IP`. Key script targets: `scripts/deploy-config-defaults.sh`, `scripts/deploy-tailscale.sh`,
`docs/operations-runbook.md` (opens with real node IPs), `docs/developer-guide.md`, `docs/api-reference.md`, `docs/hotfix-process.md`.
4. Fix the audit tool that let this happen: `scripts/audit-secrets.sh:28` — remove `\.md$` and
bare `test` from ALLOW_PATTERNS; add `sk-ant-` and password-table patterns; scan all
tracked files not just `*.env`. Run it clean as a Phase-1 exit check.
5. `.gitignore` additions: `.claude/`, `*.key`, `*.pem`, `id_rsa*`, `*.sqlite`, `*.db`
(`.claude/settings.local.json` with creds is currently only ignored by a machine-global rule).
6. Product-security note to raise (not fix now): `password123` is a shipped default (auth.rs, en.json, user-walkthrough) — file a public issue for forced first-run password change if not already enforced.
## Phase 2 — Repo restructure: deletions, binaries, layout
Delete (each its own commit):
- `loop/` (AI overnight harness w/ node SSH lines), `.agents/`, `.codex`, `.githooks/pre-push`
(the hook that re-commits the 27 MB APK — root cause of the 5.5 GB history).
- `indeedhub/` submodule + `.gitmodules` entry (points at private HTTP Gitea, breaks `--recursive`
clones); `indeedhub-demo/` (single Dockerfile — merge or drop).
- `RELEASE-NOTES-v1.0.0.md` (superseded by CHANGELOG), `neode-ui/docs/GAMEPAD-NAV-MAP.md` (duplicate of `docs/GAMEPAD-NAV.md`).
- Stray generated HTML: `docs/container-architecture.html` (311 KB), `docs/archive/architecture-review.html`, `docs/archive/lora-functionality.html`.
- `Android/local.properties` from tracking (local absolute path); remove `Android/app/debug.keystore` (standard practice).
Move out of git (→ release assets on the Releases page, referenced by URL):
- `neode-ui/public/packages/archipelago-companion.apk` (27 MB), `wireguard.apk` (17 MB), `atob.s9pk` (23 MB).
- `Android/archipelago-0.3.0-debug.apk.zip` (16 MB, stale).
- `demo/content/music/*` + heavy `demo/aiui/assets` (~261 MB, third-party/unclear-licence media — MUST not ship publicly regardless of size).
- `neode-ui/dev-dist/` (generated Workbox output) → gitignore.
Rename/fix the naming lie: `image-recipe/_archived/` contains the *production* ISO builder
(`build-auto-installer-iso.sh`, referenced by `.gitea/workflows/build-iso.yml`). Move live
files up into `image-recipe/`, delete the genuinely archived rest.
## Phase 3 — Registry domain + parameterization (functional blocker)
Infra (user assists: DNS + TLS):
- Put a domain (e.g. `registry.archipelago-os.org` / `git.archipelago-os.org`) with HTTPS in
front of the existing Gitea on vps2. OTA download URLs move from plain HTTP to HTTPS.
Repo changes:
- Introduce a single source of truth for the registry host (e.g. `REGISTRY_HOST` in
`scripts/lib/` + a default in the orchestrator config). Replace `146.59.87.168:3000` in:
all 56 `apps/*/manifest.yml`, `app-catalog/catalog.json`, `releases/manifest.json`,
`release-manifest.json`, the 11 scripts (`self-update.sh`, `create-release.sh`,
`generate-app-catalog.sh`, `validate-app-manifest.sh`, `first-boot-containers.sh`, …),
both `demo-images.yml` workflows, `demo-deploy/.env.example`, and the Android sources
(`FipsPreferences.kt`, `PartyScreen.kt`).
- Because the catalog is signed: regenerate + re-sign + republish the app catalog after the
manifest host change (catalog-overlay supremacy — disk edits don't apply otherwise).
Signing needs the user's mnemonic → schedule one ceremony after manifests are final.
- Verify: fresh machine with no LAN/tailnet access can `podman pull` one app image via the
domain and the gate node still installs apps after the re-signed catalog lands.
## Phase 4 — Documentation overhaul
### 4a. Community/legal files (missing today)
- `LICENSE` — MIT (matches existing README badge). Add `[workspace.package] license` +
`license.workspace = true` in the 5 member Cargo.tomls (also see Phase A4).
- `SECURITY.md` — disclosure address, PGP key, supported-versions; cite the March 2026 audit (`docs/archive/security-code-audit-2026-03.md`).
- `CODE_OF_CONDUCT.md` — Contributor Covenant (CONTRIBUTING.md already links to it, 404 today).
- `CONTRIBUTING.md` edits: Gitea→GitHub fork flow, remove private deploy instructions, absorb
the public-worthy CLAUDE.md invariants (rootless podman, manifest-driven, secrets model,
non-destructive migrations), versioning policy note for the `-alpha` scheme.
- `CLAUDE.md` — rewrite: keep invariants/build-verify (public-worthy), remove status banner,
node numbers, `gitea-ai` push mechanics, MEMORY references (those move to private notes).
### 4b. New developer docs (the three real gaps for app developers)
1. **`docs/quadlet-compilation.md`** — how a manifest becomes a Quadlet/systemd unit: naming,
`systemctl --user` lifecycle, where units land, how to inspect/debug one. (Source:
`core/archipelago/src/container/quadlet*.rs`, prod_orchestrator.)
2. **`docs/container-lifecycle.md`** — the 30 s level-triggered reconciler, install/adopt/
restart/uninstall state machine, health checks, crash recovery. (Replaces the plan-shaped
`docs/bulletproof-containers.md` as the current description; salvage its content.)
3. **`docs/secrets.md`** — `generated_secrets` declaration → materialisation by
`container::secrets` (0600, rootless) → injection; what developers must never do.
- Also: make every example in `docs/app-developer-guide.md` + `apps/*/manifest.yml` copy-paste
work against the new public registry host; add an end-to-end "write your first app" walkthrough
that a stranger can follow with only the public repo + an Archipelago node.
### 4c. Sanitize-and-keep internal docs (user's transparency choice)
- Keep, after Phase-1 scrubbing: `docs/PRODUCTION-MASTER-PLAN.md`, `docs/UNIFIED-TASK-TRACKER.md`,
`docs/1.8.0-RELEASE-HARDENING-PLAN.md`, `docs/RETICULUM-TRANSPORT-PROGRESS.md`, HANDOFF-*, test
plans, `docs/archive/*` — but **move all session/handoff/tracker material under
`docs/history/`** (extending the existing honest `docs/archive/README.md` pattern) so the
top-level `docs/` reads as current reference only. Add a banner to each: "historical working
document, sanitized; not maintained."
- Remove dangling agent-memory references in tracked docs (`docs/bulletproof-containers.md`,
`docs/RETICULUM-TRANSPORT-PROGRESS.md`, `docs/registry-manifest-design.md`,
`docs/bitcoin-multi-version-design.md` progress block).
- De-status the 14 design docs (strip "Status/RESUME POINT" headers into a one-line status
field; e.g. `docs/APP-PACKAGING-MIGRATION-PLAN.md` → public app-platform design doc).
- Extract North-Star narrative from PRODUCTION-MASTER-PLAN into `docs/ROADMAP.md`; extract
the "run the gate ON the node" philosophy from `docs/multinode-testing-plan.md` into
`tests/lifecycle/TESTING.md`.
- Add `docs/README.md` index (bitcoin/bitcoin `doc/` style): Getting started / Architecture /
App development / Operations / Design docs (ADRs) / History.
- README fixes: LICENSE link becomes real, Documentation table repointed at the reorganized
docs, remove "Deploy to a Test Node" private-LAN section, point Contributing at
CONTRIBUTING.md only.
## Phase 5 — Deep code cleanup (ordered zero-risk → highest-risk; cut-line after any commit)
### A. Zero-risk deletions & metadata (S each, own commits)
- **A1** Delete orphan non-compiling StartOS crates: `core/models`, `core/helpers`,
`core/js-engine` (incl. 2 committed `JS_SNAPSHOT.*.bin`), `core/container-init` (~4,100 LOC,
zero references). Verify: `cargo build --workspace && cargo test --all-features`.
- **A2** Delete unreferenced Vue components: `neode-ui/src/components/{AppSwitcher,EmptyState,SkeletonCard}.vue`. Verify: `npm run type-check && npm run build`.
- **A3** Fix `.gitignore` lockfile lines (7: `Cargo.lock`, 15: `package-lock.json`) — lockfiles are intentionally tracked; the rules are misleading and swallow future lockfiles.
- **A4** LICENSE + Cargo license fields (see 4a). Verify with `cargo metadata`.
- **A5** `core/rust-toolchain.toml` pinning `1.95.0`; align `.github/workflows/ci.yml` (remove explicit `toolchain: stable` input so the file wins). Upgrades become deliberate PRs.
- **A6** `core/rustfmt.toml` codifying **defaults only** (`edition = "2021"` + comment) — do NOT add style options days before launch (whole-tree reformat churn). Verify `cargo fmt --all -- --check` yields no diff.
### B. CI guards (zero runtime risk)
- **B1** Enable vitest in CI: run `cd neode-ui && npm run test` locally; fix trivial failures, `.skip`+issue flaky ones; add step to the frontend job. Playwright → tracked issue only (needs browsers + mock backend orchestration).
- **B2** Raw podman/systemctl **ratchet, not migration**: the 132 raw `Command::new("podman"/"systemctl")` sites use subcommands the `core/container/src/podman_client.rs` wrapper doesn't expose (network/inspect/ps/port), 43 sites are in gate-critical `install.rs`, and the prod path intentionally uses Quadlet+systemctl. Add `scripts/ci/raw-podman-ratchet.sh` (count vs committed baseline, fail on increase) as a CI step + tracked issue for wrapper API design.
### C. Clippy suppression trim (`core/archipelago/src/main.rs:8-18`, per-lint commits)
- Remove cheaply: `assertions_on_constants`, `drop_non_drop`, `wildcard_in_or_patterns`, `doc_lazy_continuation`, `enum_variant_names` (targeted allows on serde enums — never rename wire variants).
- Own careful commit: `unused_io_amount` — a **correctness** lint; fix sites with `read_exact`/`write_all` or documented targeted allows (`mesh/serial.rs:456,496` has raw partial reads; serial framing may be intentional). Full test suite + gate after.
- Keep crate-wide with justifying comment: `too_many_arguments`, `type_complexity`; attempt `ptr_arg` (`&Vec<T>`→`&[T]`, mechanical) if time allows — first to cut.
- Verify each: `cargo clippy --all-targets --all-features -- -D warnings && cargo test --all-features`.
### D. dead_code lift — Tiers 1–2 pre-launch, Tier 3 → commented allows + issues
Per-module procedure (one file per commit): remove `#![allow(dead_code)]` → `cargo check
--all-targets --all-features` → triage each warning: (a) genuinely dead → delete;
(b) future-feature/protocol-mandated → targeted `#[allow(dead_code)] // TODO(#NNN): …`;
(c) missing wiring → keep + targeted allow + issue (don't fix wiring in this workstream) →
clippy `-D warnings` + tests → commit.
- **Tier 1 (small/leaf, S each):** `swarm/seed_advert.rs`, `transport/{mesh_transport,lan,chunking,delta}.rs`, `mesh/{crypto,alerts,types,outbox}.rs`, `streaming/mod.rs`, `wallet/mod.rs`.
- **Tier 2 (M each):** `fips/{mod,iface,dial}.rs` (41 external refs → little residual deadness), `mesh/{x3dh,ratchet,steganography,message_types}.rs` — for crypto files bias to (b) with roadmap comments (unused crypto attracts auditor noise; every kept item needs its why).
- **Tier 3 (defer, riskiest):** `mesh/{mod,reticulum,protocol,serial,bitcoin_relay}.rs`, `transport/mod.rs` — change each blanket allow to `#![allow(dead_code)] // Hardware-mesh surface partially wired; triage tracked in #NNN`.
- Optional S/M win: move `prod_orchestrator.rs`'s 5,034-line `#[cfg(test)]` module to a sibling file via `#[path]` (pure move, halves the 6,291-line file).
### E. stacks.rs legacy fallbacks (highest risk — LAST, evidence-gated)
Legacy installers for immich/btcpay/mempool/indeedhub (`core/archipelago/src/api/rpc/package/stacks.rs:838/1047/1267/1498`, ~1,000 LOC with hardcoded registry IPs) fire only on "unknown app_id, zero members installed", logging `INSTALL ORCH SKIP` (stacks.rs:673). Netbird already uses the hard-error replacement (stacks.rs:1898-1920).
1. Run the full gate on the node; grep install logs for `INSTALL ORCH SKIP`.
2. Zero SKIPs → replace each legacy body with the netbird-style hard error (keep orchestrator call + `adopt_stack_if_exists`; satisfies migrations-never-destroy-data). Re-run gate; any red → revert + issue.
3. Any SKIP → don't delete; issue: "deploy manifests fleet-wide, then delete legacy installers".
### Explicitly deferred → public tracked issues at launch
PodmanClient API extension + call-site migration; god-module splits (`install.rs`, `update.rs`, `mesh/mod.rs`); Playwright in CI; Tier-3 dead_code triage; `password123` default hardening.
## Phase 6 — Fresh-history publish
1. Freeze: all phases merged on internal `main`, gate green, catalog re-signed.
2. Build the public tree: `git archive`-style export of HEAD (never copy `.git/` — it holds
credentialed remotes) → new repo, single initial commit ("Initial public release, vX.Y.Z"),
optionally preserving CHANGELOG.md as the human-readable history.
3. Pre-publish gate on the export: `scripts/audit-secrets.sh` (fixed version) clean; grep-zero for
`sk-ant-`, rotated-password strings, `146.59.87.168`, tailnet `100.` IPs, `192.168.1.`,
internal hostnames; `du -sh .git` sanity (< ~100 MB); fresh `git clone` + `cd core && cargo build`
+ `cd neode-ui && npm ci && npm run build` on a clean machine/container; one app image pull
from the public domain.
4. Publish to GitHub; enable issue templates (already present in `.github/`); file the deferred-work
issues (from Phase 5's issue list) as the initial public issue set — honest and gives contributors entry points.
5. Internal repo remains the private full-history remote; decide sync direction post-launch
(recommend: public repo becomes canonical, private keeps only ops/infra notes).
## Verification (end-to-end)
- `tests/lifecycle/run-gate.sh` green on the node after Phases 3 + 5E (and after any lifecycle-touching commit).
- CI green on every phase commit: `cargo fmt --check`, `clippy -D warnings`, `cargo test --all-features`, frontend type-check + build + (new) vitest.
- Phase-6 clean-machine clone/build/pull test is the final acceptance test — it simulates the first outside developer.
- Docs acceptance: a reader following `docs/app-developer-guide.md` + the new quadlet/lifecycle/secrets docs can build and install an app manifest without any private infra.
## Sequencing / cut-line
Order: 0 → 1 → 2 → (3 ∥ 4) → 5 (A→E) → 6. Phases 0–2 are non-negotiable security; Phase 3 is the
functional blocker; Phase 4 is the developer-experience payload; Phase 5 can be cut after any
commit (minimum viable: A1–A6, B1–B2, unused_io_amount fix); Phase 6 last. If the timeline
compresses, Tier-2 dead_code and Phase E move to public issues — everything else holds.
docs/PRODUCTION-MASTER-PLAN.md
-1265
View File
File diff suppressed because it is too large Load Diff
docs/README.md
+89
View File
@@ -0,0 +1,89 @@
# Archipelago documentation
Start here. This index groups the docs by what you're trying to do. The
authoritative behaviour is always the code in `core/`; where a doc and the code
disagree, the code wins and the doc is a bug.
## Getting started
- [User Walkthrough](user-walkthrough.md) — setting up and using a node, from hardware to daily use
- [Talking to your node](COMMANDS.md) — the conversational command surface
- [Seed Verification](SEED-VERIFICATION.md) — independently verify your 24-word backup
- [Troubleshooting](troubleshooting.md) — common problems and how to resolve them
- [Gamepad / Controller Navigation](GAMEPAD-NAV.md) — driving the UI from a controller
- [Pine voice commands](pine-voice-commands.md) — the voice-satellite phrase surface
## Architecture
- [Architecture](architecture.md) — the system at a glance
- [Multi-Node Architecture](multi-node-architecture.md) — how nodes relate across a fleet
- [API Reference](api-reference.md) — the JSON-RPC surface
## Contributing to Archipelago itself
- [Developer Guide](developer-guide.md) — building the workspace, the frontend, and an ISO
- [Contributor guide (`CLAUDE.md`)](../CLAUDE.md) — invariants, build/verify, the production test gate
- [Bulletproof containers](bulletproof-containers.md) — why the reconciler is level-triggered
- [Release signing runbook](workstream-b-signing-runbook.md) — the ceremony and key handling
- [1.8.0 Release Hardening Plan](1.8.0-RELEASE-HARDENING-PLAN.md) — the release-blocking checklist
- [Third-party license audit](LICENSE-COMPLIANCE-AUDIT.md) — dependency licensing posture
- [Demo build info](demo-build-info.md) — operating the public demo sandbox
## App development
- [App Developer Guide](app-developer-guide.md) — build and package a containerized app
- [App Manifest Specification](app-manifest-spec.md) — the manifest schema, field by field
- [Manifest → Quadlet unit](quadlet-compilation.md) — how a manifest compiles to a systemd-owned container unit
- [Container lifecycle](container-lifecycle.md) — the reconciler state machine: install/adopt/start/stop/self-heal
- [App secrets](secrets.md) — declaring, generating and injecting per-install credentials
- [Registry-Distributed Manifests](registry-manifest-design.md) — how manifests reach nodes via the signed catalog
- [Decentralized Marketplace Protocol](marketplace-protocol.md) — publishing apps via an external registry
- [Bitcoin RPC Relay](bitcoin-rpc-relay.md) — letting an external wallet reach the node's Bitcoin RPC
- [Companion Pairing QR](companion-pairing-qr.md) — the pairing handoff contract
- [TV input inside iframe apps](tv-input-iframe-apps.md) — keyboard/gamepad routing into embedded apps
## Design docs
These record why a thing is built the way it is. They are design records, not
step-by-step guides, and some predate the current implementation.
- [Registry-Distributed Manifests](registry-manifest-design.md)
- [DHT Distribution](dht-distribution-design.md)
- [Bitcoin Multi-Version](bitcoin-multi-version-design.md)
- [Dual Ecash](dual-ecash-design.md)
- [Hardware Signer](hardware-signer-design.md)
- [Manifest Hooks](manifest-hooks-design.md)
- [Meshroller Integration](meshroller-integration-design.md)
- [Nostr Git Source Hosting](nostr-git-source-hosting.md)
- [Nostr Identity Import](nostr-identity-import-plan.md) · [Nostr Signer Login (research)](nostr-signer-login-research.md)
- [Streaming Ecash (phase 4)](phase4-streaming-ecash-plan.md)
- [App Packaging Migration](APP-PACKAGING-MIGRATION-PLAN.md)
## Decisions (ADRs)
- [ADR-001: Podman over Docker](adr/001-podman-over-docker.md)
- [ADR-002: DID Key Method for Node Identity](adr/002-did-key-method.md)
- [ADR-003: Nostr Relays for Discovery](adr/003-nostr-for-discovery.md)
- [ADR-004: Tor Hidden Services for Peer Communication](adr/004-tor-for-peer-communication.md)
- [ADR-005: ChaCha20-Poly1305 for Backup Encryption](adr/005-chacha20-backup-encryption.md)
- [ADR-006: Nostr Relays for Marketplace Discovery](adr/006-nostr-marketplace-discovery.md)
- [ADR-007: DID-Based Federation Trust](adr/007-did-federation-trust.md)
- [ADR-008: Dual Key Strategy (Ed25519 + Secp256k1)](adr/008-dual-key-strategy.md)
- [ADR-009: Manifest-Level Container Security](adr/009-manifest-container-security.md)
- [ADR-011: DWN Deprioritization](adr/011-dwn-deprioritization.md)
There is no ADR-010 — the number was never issued, so the gap is not a missing
file.
## Security
- [Security Policy](../SECURITY.md) — how to report a vulnerability
- [PSBT Signing Architecture](security/PSBT-SIGNING-ARCHITECTURE.md)
- [Bitcoin RPC Proxy Exposure](security/BITCOIN-RPC-PROXY-EXPOSURE.md)
- [Entropy Enforcement (KEY-05)](security/KEY-05-ENTROPY-ENFORCEMENT.md)
## Roadmap & history
- [Roadmap](ROADMAP.md) — where the project is going
- [archive/](archive/README.md) — superseded design and status documents, kept for provenance
docs/RETICULUM-TRANSPORT-PROGRESS.md
-399
View File
@@ -1,399 +0,0 @@
# Reticulum mesh transport — progress tracker
Living status doc for the Reticulum (RNS+LXMF) third-transport work. **Update this after every
meaningful step.** If a session is cut off mid-work, read this file first, then the plan, then
resume at "Next up."
Full plan: `.claude/plans/enchanted-strolling-rocket.md`. Memory pointer:
`project_reticulum_transport_plan.md` (auto-memory index).
**Coordination note (2026-06-30):** a separate agent owns concurrent Meshtastic work, scoped to
`mesh/meshtastic.rs` + `mesh/protocol.rs` (see `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md`) and explicitly
avoiding `mesh/listener/session.rs` transport plumbing + `mesh/mod.rs` routing, which this work
owns. Stay out of `meshtastic.rs`/`protocol.rs` to avoid collisions.
## Checkpoint 2026-07-28 — RNode connect + names FIXED, live-verified E2E (read this first)
The fleet reflash back to RNode firmware exposed a stack of bugs that made Reticulum
unusable on CP2102-bridged boards (Heltec V3 etc.) and left every archy node nameless on
RNS. All fixed in `a8c4694c` (backend) + `3f76b496` (UI), live-verified on archi-dev-box
and archy-x250-dev with a real RNode-to-RNode LXMF message (`transport: "reticulum"` in
mesh-messages) plus a cross-transport reply:
1. **probe_rnode boot race** — serial open pulses DTR/RTS via the USB-UART bridge → ESP32
power-cycles → KISS DETECT written 300ms later is eaten during ~2.5-3s of boot. Fix:
immediate probe (fast path) + drain-until-quiet boot settle + second DETECT window.
2. **configure() was a no-op on a running listener** (only enable/disable restarted it) —
the setup modal's apply/keep-as-is and every rename did nothing until process restart.
3. **Name propagation** — `config.advert_name` had no reader; `server.set-name` never
reached mesh; daemon display name fixed at spawn to the "Archy" default; the ARCHY:2
announce blob REPLACED the LXMF name. Now: announces carry msgpack
`[name, stamp_cost, sf, ARCHY-blob]` (Sideband-compatible, blob invisible to stock
clients), daemon has a `set_name` verb, renames bounce the session live.
4. **Daemon-death detection** (was invisible up to the 30-min RX-stall watchdog),
**modal re-trigger loop** (plugged_at used tty mtime → bumps on every open; now
btime/ctime), **ARCHY:2 federation-name clobber**, **mesh.refresh RPC** (Refresh button
now actually re-queries the radio), **Meshtastic mesh.broadcast now sends NodeInfo**.
Still open here: legacy-format peers (old fleet builds) show as `Reticulum <hex4>` until
they OTA; RNode RF params still daemon-hardcoded (EU-868 869.525/125k/SF8/CR5); Phase 4
multi-radio; duty-cycle guard.
## Status at a glance
| Phase | What | Status |
|---|---|---|
| 0 | Gate #1 — deterministic identity from Archy keys | ✅ **DONE**, verified in venv AND in the PyInstaller binary (same dest hash) |
| 0 | Gate #2 — two-node LXMF-over-LoRa on real hardware | ✅ **PASSED 2026-06-30** — real RF announce + encrypted DM exchanged between .116's Heltec V3 RNode and a phone-flashed second RNode running Sideband |
| 0 | Gate #3 — external Sideband/MeshChat interop | ✅ **PASSED 2026-06-30** — same session as gate #2; Sideband is the stock external client this gate calls for |
| 1 | `reticulum-daemon/` (Python rns+lxmf, Unix-socket RPC) | ✅ scaffolded + tested (no radio); signed-identity announce **also done** (see below) |
| 1 | Packaging — PyInstaller single binary | ✅ **DONE + verified** — `reticulum-daemon/build.sh`, 16M standalone binary, selftest passes run from `/tmp` with no venv on PATH |
| 2 | Rust wiring (`DeviceType`, `MeshRadioDevice`, `ReticulumLink`, stamp sites) | ✅ **`cargo check`/`cargo test -p archipelago` GREEN** (99 mesh tests pass) — still untested on real hardware |
| 2c | `MeshConfig.device_kind` reflashable-board pin | ✅ **DONE** this session (was the one open Phase-2 item) |
| 3 | Frontend (~8 label/CSS spots) | ✅ DONE (scoped down — see note below) |
| 4 | Multi-device (run all 3 radios at once) + per-network channels | ⏳ not started (follow-on, after 0–3) |
| 5 | Aurora interop — optional plain-TCP Reticulum interface (radio-less) | ✅ **DONE + verified 2026-07-03** — see checkpoint below. Real Aurora GUI test still open (manual follow-up). |
## Checkpoint 2026-06-30 (late session — read this first if cut off)
This session picked up after Phase 2/3 were already green, and closed out everything that didn't
need real RNode hardware:
1. **Corrected two stale tracker entries** (both were already done, just not reflected here):
- The `_announce_app_data` "TODO" was actually already implemented:
`reticulum_daemon.py`'s `_announce_app_data()` embeds `ARCHY:2:{ed}:{x25519}` when
`--archy-ed-pubkey-hex`/`--archy-x25519-pubkey-hex` are passed, and `reticulum.rs`'s
`daemon_command()`/`open()` already forward `our_ed_pubkey_hex`/`our_x25519_pubkey_hex` from
`session.rs` (`run_mesh_session` → `auto_detect_and_open`/`open_preferred_path` →
`ReticulumLink::open`). Confirmed end-to-end by reading the call chain, not just grepping.
- Phase 3 frontend was already done (see prior entry below) — tracker table above said
"not started", now corrected.
2. **Added `MeshConfig.device_kind: Option<DeviceType>`** (plan §2c, the one explicitly-listed
open Phase-2 item) — `mesh/mod.rs` (field + Default + threaded into `start()`'s
`spawn_mesh_listener` call), `listener/mod.rs` (`spawn_mesh_listener` param → `run_mesh_session`
arg), `listener/session.rs` (`run_mesh_session` param; `auto_detect_and_open` skips
non-matching probes per-path via `device_kind.is_none_or(|k| k == ...)`;
`open_preferred_path` restructured to a `match kind { ... }` that tries **only** the pinned
driver and surfaces its real error, instead of silently falling through to another firmware's
handshake on the same port). `None` (default) preserves today's strict
Meshcore→Meshtastic→Reticulum auto-detect — fully backward compatible, no config migration
needed. `cargo check` + `cargo test -p archipelago` both green after (99 mesh tests, 0 failed).
3. **Built and verified the PyInstaller packaging** (plan's Phase 1 "Packaging" + the file list's
"Ops: release packaging to include the daemon binary" item — previously undone):
- `reticulum-daemon/build.sh` (new) — reproducible build, installs `requirements-build.txt`
(new, `pyinstaller==6.21.0`, build-only/not shipped) into the existing `.venv`, runs
PyInstaller with flags discovered by trial: `--collect-submodules RNS --collect-submodules
LXMF --collect-data RNS -d noarchive`.
- **Non-obvious gotcha, written up in `build.sh`'s comments so it isn't re-discovered:**
`RNS.Interfaces/__init__.py` builds its `__all__` via `glob.glob(os.path.dirname(__file__) +
"/*.py")` at import time (`Reticulum.py` does `from RNS.Interfaces import *`). PyInstaller's
default `--onefile` zips pure-Python modules into an in-binary PYZ archive, so `__file__`
doesn't point at a real directory and the glob comes back empty → `NameError: name
'Interface' is not defined` the moment `RNS.Reticulum(...)` is constructed. `-d noarchive`
(keep modules as loose `.pyc` files on disk inside the onefile bundle's runtime-extraction
dir) fixes it — confirmed by reproducing the failure first, then fixing it.
- **Verified, not just built:** ran the resulting `dist/archy-reticulum-daemon` binary's
`--check` (dest hash matches the venv-derived `06bb31e16f4f8d46a8ae8eac23a4fd21` for the
test seed) and `--selftest` (full RNS+LXMF bring-up, no radio) **both from `/tmp` with the
binary copied away from the repo and the `.venv` not on `PATH`** — confirms it's genuinely
self-contained, not accidentally still depending on the dev venv.
- `dist/`/`build/`/`*.spec` are already gitignored (`reticulum-daemon/.gitignore`); only
`build.sh` + `requirements-build.txt` are new tracked files.
**NOT done this session (still genuinely open):**
- Everything hardware-dependent (Phase 0 gates #2/#3, real RNode probe/spawn). The .116 Heltec V3
reflash mentioned in the prior session's memory was **not** done in this session — no physical
hardware access was exercised, only software.
- `/dev/reticulum-radio` udev symlink (plan §2c) — **deliberately not added**: the existing
`99-mesh-radio.rules` keys on USB vendor/product ID (e.g. CP2102 0x10c4/0xea60), but the whole
point of `device_kind` is that the *same* chip can run any of the three firmwares — a
vendor/product udev rule can't disambiguate them, and a fabricated rule would just be
misleading. Real fix needs either a per-device `ATTRS{serial}==...` rule the operator fills in
once they know their specific board's serial (no such board exists in-repo to template from
yet), or rely on `device_kind` alone (already done, works regardless of `/dev` path naming).
Revisit once a real RNode-flashed board's serial is known.
- PyInstaller binary not yet wired into the release tarball / `scripts/deploy-to-target.sh` (the
daemon binary path is currently resolved via `ARCHY_RETICULUM_DAEMON_BIN` env or the dev venv
fallback in `reticulum.rs`'s `daemon_command()` — production default
`/usr/local/bin/archy-reticulum-daemon` is a real path convention now that `build.sh` produces
exactly that filename, but nothing copies it there yet). Left undone deliberately — wiring
release-tarball plumbing for a binary that's never been run against real RNS network traffic
felt premature; do this once Phase 0 gates #2/#3 pass.
## Phase 2 — Rust wiring detail (what's done vs left)
**Done — `cargo check -p archipelago` is GREEN:**
- `core/archipelago/src/mesh/types.rs` — `DeviceType::Reticulum` (+ `Display` arm) + a
`radio_transport_label(DeviceType) -> &'static str` helper (`"reticulum"` vs `"lora"`).
- `core/archipelago/src/mesh/mod.rs` — all 4 outbound stamp sites use
`radio_transport_label(...)`; `use_typed_envelope` (~1571) extended to
`matches!(device_type, Meshcore | Reticulum)`; `data_dir` threaded into
`spawn_mesh_listener(...)` call (was: `MeshService::start()` → `spawn_mesh_listener`).
- `core/archipelago/src/mesh/listener/mod.rs` — `spawn_mesh_listener` takes `data_dir:
PathBuf`, passes `&data_dir` into `run_mesh_session`.
- `core/archipelago/src/mesh/listener/decode.rs:406,639` and `dispatch.rs:79` — all 3 inbound
stamp sites now use `radio_transport_label(state.status.read().await.device_type)`.
- `core/archipelago/src/mesh/listener/session.rs`:
- `MeshRadioDevice` enum has `Reticulum(ReticulumLink)`; all 18 method arms wired (no-ops:
`ensure_lora_region`, `ensure_channel`, `send_keepalive`, `send_nodeinfo_advert`, `reboot`,
`reset_contact_path`; everything else forwards to `ReticulumLink`).
- `auto_detect_and_open(data_dir: &Path)` and `open_preferred_path(path, data_dir: &Path)`
both now try `ReticulumLink::open(path, data_dir)` **last**, after Meshcore/Meshtastic —
cheap raw-serial KISS-detect probe runs first; the daemon only spawns on a confirmed match.
- `reticulum_contact_id()` helper added (delegates to the canonical
`reticulum::reticulum_contact_id_from_hash`, masked `& 0x7FFF_FFFF`, avoids 0).
- `refresh_contacts()` has an `is_reticulum` branch parallel to `is_meshtastic`; `reachable`
flows through `contact.path_len != 0` unchanged (`ReticulumLink::get_contacts()` already
encodes daemon-reported reachability into `path_len`).
- `data_dir: &Path` threaded through `run_mesh_session` → both probe functions.
- `core/archipelago/src/mesh/reticulum.rs` — **created**. `ReticulumLink`: spawns/supervises the
daemon as a child process, Unix-socket RPC client (matches the tested daemon contract),
`prefix_to_hash: HashMap<[u8;6],[u8;16]>` (mandatory per the plan), synthetic
`InboundFrame` builder byte-matching `meshtastic.rs`'s layout, `Drop` impl that kills the
daemon + cleans up the socket. Has unit tests (KISS-detect byte matching, contact-id masking,
synthetic-frame layout) — **passing, see below**.
**Concurrent-edit note:** a separate in-flight change (not mine) added `MeshPeer.pkc_capable`
and `ParsedContact.pkc_capable` (Meshtastic PKI-capability tracking) while this work was in
progress. Accounted for: `reticulum.rs`'s `ParsedContact` literal sets `pkc_capable: false`
(Reticulum/LXMF is unconditionally E2E via `take_rx_encrypted()`, this field has no analogue);
two incomplete `MeshPeer` literals in `decode.rs` (lines ~330, ~548) were completed with
`pkc_capable: false` to unblock the build for everyone — not reverted, not worked around.
**Self-review fix applied:** the RPC Unix socket originally lived in the shared system temp
dir; moved to `{data_dir}/reticulum/` (0700) instead — archipelago-owned, not shared `/tmp`,
matching the security posture. Re-confirmed `cargo check -p archipelago` GREEN after the move.
**NOT yet done:**
- `MeshConfig.device_kind: Option<DeviceType>` hint (optional reflashable-board disambiguator,
plan §2c) — not added. Auto-detect ordering (Meshcore→Meshtastic→Reticulum, strict probes)
is the only disambiguator right now.
- Phase 3 frontend — **DONE**, but **smaller scope than originally inventoried**: only
`Mesh.vue`'s `transportLabel()` (per-message field) + `mesh-styles.css` `.transport-reticulum`
+ the `mesh.ts` doc comment needed the addition. `transport.ts` `TransportKind`,
`federation/types.ts` `last_transport`, `NodeList.vue` `transportBadge`, and `PeerFiles.vue`
`transportPill` are a COARSER routing-layer category (`mesh`/`lan`/`fips`/`tor`) where
`'mesh'` already covers any radio (meshcore/meshtastic/reticulum) — adding a separate
`'reticulum'` there would be inconsistent with how meshcore/meshtastic are handled. Confirmed
via `vue-tsc --noEmit` (exit 0, zero errors).
- Everything hardware-dependent: real daemon spawn/probe against an actual RNode (the .116
Heltec V3, once reflashed), two-node LXMF-over-LoRa, the `_announce_app_data` signed-identity
TODO in the daemon (currently carries only the plaintext display name, not a verified Archy
DID/pubkey — needed for `bind_federation_twins`-style auto-binding across protocols).
## Verified facts to reuse (don't re-derive)
**RNode KISS-detect handshake** (confirmed against the canonical Reticulum source, not guessed):
```
constants: FEND=0xC0 FESC=0xDB TFEND=0xDC TFESC=0xDD CMD_DETECT=0x08 DETECT_REQ=0x73 DETECT_RESP=0x46
probe tx: C0 08 73 C0 50 00 C0 48 00 C0 49 00 C0 (detect + fw_version + platform + mcu queries)
success: response contains byte sequence ... C0 08 46 ... (FEND, CMD_DETECT, DETECT_RESP)
```
Source: `RNS/Interfaces/RNodeInterface.py` (Liberated Systems mirror), `detect()`/`readLoop()`.
**Synthetic `InboundFrame` layout** for a 1:1 DM, copied exactly from
`meshtastic.rs:1031-1047` (`ReticulumLink` must build the same shape so `frames::handle_frame`
needs zero changes):
```
data = [snr(1)=0][reserved(2)=00,00][sender_prefix(6)][path(1)=0xff][type(1)=0][rx_time(4 LE)][payload…]
code = RESP_CONTACT_MSG_V3_E2E if encrypted else RESP_CONTACT_MSG_V3 (RNS/LXMF is always E2E, so always _E2E)
```
Channel/broadcast equivalent (`RESP_MESHTASTIC_CHANNEL_TEXT`, meshtastic.rs:1019-1028) — N/A for
Reticulum in single-device Phase 2 (LXMF has no shared-channel concept); revisit in Phase 4.
**`resolve_peer`** (decode.rs:316) matches inbound `sender_prefix` against
`peer.pubkey_hex.starts_with(prefix)` — so as long as `refresh_contacts`/announce-handling
populates `pubkey_hex` = full 16-byte RNS hash hex BEFORE a message arrives (same precondition
meshtastic relies on via its `peer_pubkeys` map), no Reticulum-specific fallback is needed there.
**`ParsedContact.public_key_hex`** for Reticulum = hex of the 16-byte RNS dest hash (32 hex
chars, NOT 32 bytes) — the `hex::decode(...).len()==32` checks elsewhere (e.g. the auto-heal
`reset_contact_path` loop in `refresh_contacts`) will naturally skip Reticulum contacts since
their key decodes to 16 bytes, not 32. That's fine — no special-casing needed, just don't "fix"
it to be 32 bytes.
**`data_dir.join("identity").join("node_key")`** is the 32-byte raw Ed25519 seed file — this is
exactly what `reticulum_daemon.py --identity-key <path>` expects (confirmed against
`identity.rs` `NODE_KEY_FILE`/`load_or_create`). The daemon reads the file itself — Rust should
pass the **path**, not pipe the raw key bytes through more hops than already exist.
## Hardware update (2026-06-30)
**.116 has a Heltec V3 available to reflash with RNode firmware.** This unblocks Phase 0 gates
#2/#3 (previously marked blocked — `.198`'s radio is dead, but .116's Heltec V3 is a real path
forward without needing new hardware). Next concrete step once reflashed: run
`reticulum-daemon/reticulum_daemon.py` pointed at the RNode's serial path, confirm `--check`
hash matches `--selftest`, then bring up two instances (.116 + .228, after .228 also gets an
RNode-capable board) for the real two-node LXMF-over-LoRa gate.
## Daemon contract (already built + tested — Phase 2 codes against this, no changes needed)
`reticulum-daemon/reticulum_daemon.py`, RPC over Unix socket (0600), one JSON object per line:
- in: `{"cmd":"send","dest_hash":hex16,"content":...}` / `{"cmd":"announce"}` /
`{"cmd":"status"}` / `{"cmd":"shutdown"}`
- out: `{"event":"ready",...}` / `{"event":"recv",...}` / `{"event":"announce",...}` /
`{"event":"delivered",...}` / `{"event":"status",...}`
Verified: `--check` (hash only), `--selftest` (boots real RNS+LXMF, no radio), and a live
socket round-trip (`ready`→`status`→`shutdown`, clean exit) — see `reticulum-daemon/README.md`.
## Checkpoint 2026-06-30 (hardware session — gates #2/#3 PASSED)
Picked up after a session pipe-break; the live system (archipelago.service + the spawned
`archy-reticulum-daemon`) had kept running uninterrupted the whole time, so nothing was lost.
**What happened, in order:**
1. .116's Heltec V3 (CP2102, USB vendor/product `10c4:ea60`, serial `0001`) was reflashed with
RNode firmware and plugged into `/dev/mesh-radio` (generic udev symlink → `ttyUSB0`, not a
per-serial rule). `mesh-config.json` has `device_path: null` — pure auto-detect, no
`device_kind` pin needed.
2. Auto-detect correctly tried Meshcore → Meshtastic → Reticulum and found it: journal shows
`Found Reticulum (RNode) device via auto-detect path=/dev/mesh-radio` — but only **after**
~4 min of `Failed to spawn reticulum-daemon — is it installed/packaged?` retries, because
`/usr/local/bin/archy-reticulum-daemon` hadn't been copied into place yet from
`reticulum-daemon/dist/` (built via `./build.sh`). Once copied (sha256-verified match to the
`dist/` build), auto-detect succeeded on the very next retry.
3. `mesh.status` RPC confirmed live: `device_type: "reticulum"`, `device_connected: true`,
`dest_hash: 5d146f6e1c9707f89468b5016ed6dfad`. Periodic self-advert (`send_self_advert` →
`{"cmd":"announce"}` → real RNS `Identity.announce()`) firing every ~30s — confirmed this is
**not** the `send_nodeinfo_advert` no-op arm (that one's still legitimately a no-op for
Reticulum; the real announce path is `send_self_advert`, wired correctly).
4. Second RNode flashed onto a phone running **Sideband**. First attempt showed RF energy
(`interference_last_dbm` climbing) but `rxb: 0` — a parameter mismatch, **not** a frequency
problem (energy was detected, just not demodulated). Root cause: Spreading Factor mismatch
in Sideband's manual RNode interface config (frequency display rounds to one decimal so
"869.5" silently passed at first glance — bandwidth/SF/CR are separate fields and SF was
wrong). Once SF was corrected to match (freq `869525000`, BW `125000`, **SF `8`**, CR `5`),
`rxb` went non-zero immediately and a real `{"event":"announce","dest_hash":"1870744d...",
"app_data":"7a617a61"}` (hex for "zaza") arrived over the air.
5. **Gate #2 + gate #3 both passed in the same exchange**: `zaza` shows up as a real, reachable
`mesh.peers` contact; an inbound encrypted LXMF message ("Yoooo") arrived and was correctly
stamped `encrypted: true, transport: "reticulum"`; a reply was sent back and round-tripped.
Sideband is exactly the stock external client gate #3 calls for, so one real RNode-to-RNode
LoRa link covered both gates — no need for a second dedicated archy node.
6. **Two real bugs found from this, both fixed:**
- `record_sent_typed`'s `encrypted` flag was hardcoded `false`/`archy || pkc_capable` on the
Reticulum send path (both the native-text path in `send_message` and the typed-envelope
path in `send_typed_wire`) — correct for Meshcore/Meshtastic (where E2E really is
conditional on PKI/session state not yet threaded through), **wrong** for Reticulum: LXMF
encrypts every send to the destination identity key unconditionally, archy peer or not.
Fixed: both call sites now OR in `device_type == DeviceType::Reticulum`.
- `radio_transport_label()` collapsed Meshcore **and** Meshtastic into one generic `"lora"`
string, so the per-message pill couldn't distinguish them. User asked for 3 distinct pill
colors (Meshtastic mint, Meshcore orange, Reticulum blue) — extended the label fn to
return `"meshtastic"`/`"meshcore"`/`"reticulum"` distinctly, updated `Mesh.vue`'s
`transportLabel()` switch and `mesh-styles.css` (`.transport-meshtastic` `#3eb489`,
`.transport-meshcore` `#fb923c`, `.transport-reticulum` `#60a5fa`; kept `.transport-lora`
`#f59e0b` as a fallback for any already-stored legacy-labelled messages). `cargo check` +
`vue-tsc --noEmit` both green after.
**NOT yet done:**
- The Rust-side fix above (`encrypted` flag, transport-label split) is built but **not yet
deployed to .116's running binary** — the live daemon/auto-detect verification above was all
against the binary already running before this session's edits. Rebuild + redeploy to see the
fix live.
- `tests/lifecycle/run-gate.sh` not re-run after these mesh changes yet (project convention:
run after backend changes land).
- Multi-device (3 radios at once, Phase 4) and the release-tarball/udev-rule wiring (originally
"Next up" #6 below) are both still untouched.
## Next up (resume here)
Phase 0 gates #1–#3 are now **all passed**. What's left:
1. Rebuild the backend + frontend and redeploy to .116 so the `encrypted`-flag fix and the
3-way transport-pill color split actually take effect on the live node (currently only
checked in with `cargo check`/`vue-tsc`, not deployed).
2. Re-verify on-device after redeploy: send another Sideband↔archy DM, confirm the Sent bubble
now shows E2E + a blue "Reticulum" pill, and confirm Meshtastic/Meshcore pills (if any
messages exist) render mint/orange instead of the old generic amber "LoRa".
3. Exercise the rest of the plan's "Verification (definition of done)" items: hot-swap
detection (unplug the RNode mid-session, confirm fallback to FIPS/Tor on the same contact;
replug, confirm it picks Reticulum back up), and `device_kind: Some(Reticulum)` pin path
(currently only auto-detect has been exercised on real hardware).
4. Run `tests/lifecycle/run-gate.sh` to confirm no regression from the mesh changes landing.
5. Only after the above: wire `dist/archy-reticulum-daemon` into the release tarball /
`scripts/deploy-to-target.sh` (target path `/usr/local/bin/archy-reticulum-daemon`, matching
`reticulum.rs`'s default) and add a per-serial-number `/dev/reticulum-radio` udev rule now
that a real board's serial number (`0001` on the CP2102, .116's board) is known — though a
second board will likely report the same `0001` stock serial since CP2102 modules commonly
ship with an unprogrammed default, so this may still need a different disambiguator.
6. Phase 4 (run all 3 radios at once) — still not started, follow-on after the above.
## Checkpoint 2026-07-03 — Phase 5: Aurora interop via plain-TCP Reticulum (radio-less)
**Why:** `~/aurora` (a separate Flutter off-grid messenger) already runs real RNS + LXMF
(`LxmfRouter`, comment "interop with Sideband/NomadNet/MeshChat" in `rns_service.dart`), and its
**default** connectivity mode is plain TCP (`RnsTcpInterface`/`RnsTcpServerInterface`), not radio —
it ships a static bootstrap list of public RNS hubs on port 4242. Archy's daemon could previously
only bring up a serial-RNode interface, so it was unreachable by Aurora (or any TCP-based RNS/LXMF
client) at all, and every interop proof was bottlenecked on scarce LoRa hardware. This phase adds
an **optional, additive, loopback-only plain-TCP interface**, proves interop with a scripted
RNS/LXMF stand-in (the same class of proof the Sideband gate already established), and leaves the
serial/RNode path completely unchanged.
**Done, all verified:**
1. `reticulum-daemon/reticulum_daemon.py` — `_write_rns_config()` gained a third branch
(`--tcp-listen HOST:PORT` → `TCPServerInterface`, `--tcp-connect HOST:PORT` repeatable →
`TCPClientInterface`), mutually exclusive with `--serial-port`. `--tcp-listen` is hard-gated to
loopback (`_require_loopback`) — archy is otherwise Tor-first for inter-node traffic, so a
WAN/LAN-exposed Reticulum port is a deliberate future decision, not something this phase does
silently. Verified: `--selftest` regression still passes; two daemon processes (server +
client, throwaway identities) reached `connected: true` on both sides via `mesh.status`-daemon
RPC, live `TCPServerInterface`/`TCPClientInterface` visible in `get_interface_stats()`.
2. **Bidirectional LXMF DM gate against a scripted Aurora stand-in** (Python RNS+LXMF client
dialing as a `TCPClientInterface` + running its own `LXMRouter` — a legitimate protocol-level
proxy for Aurora's Dart stack, same wire format): forward (stand-in → archy daemon) and reverse
(archy daemon → stand-in) both delivered with matching content and correct source/dest hashes,
confirmed via the daemon's own `recv`/`delivered` RPC events. Direct TCP analogue of the
already-passed Sideband gate (RF → TCP, Sideband → scripted stand-in).
3. **Rust wiring**, fully additive — the serial/RNode path is byte-for-byte unchanged:
- `mesh/reticulum.rs`: new `ReticulumInterface` enum (`Serial`/`TcpServer`/`TcpClient`) threads
through `daemon_command()`/`spawn()`; `open()` (serial) now just wraps
`ReticulumInterface::Serial` — same `probe_rnode` gate as before. New
`open_tcp_server()`/`open_tcp_client()` associated fns skip `probe_rnode` entirely (the
"spawn without a physical RNode" path); `open_tcp_server` hard-enforces
`is_loopback_host()` (mirrors the Python-side guard).
- `mesh/types.rs`: new `ReticulumTcpConfig` enum (`Server { bind }` / `Client { connect }`).
- `mesh/mod.rs`: `MeshConfig.reticulum_tcp: Option<ReticulumTcpConfig>` (`#[serde(default)]`,
`None` by default — no migration, zero behavior change when unset); threaded into
`start()` → `spawn_mesh_listener`.
- `listener/mod.rs` / `listener/session.rs`: `reticulum_tcp` param threaded through
`spawn_mesh_listener`/`run_mesh_session`; new leading branch — if set, a new
`open_reticulum_tcp()` helper dispatches to `open_tcp_server`/`open_tcp_client`; otherwise
falls through to the **untouched** existing `preferred_path`/`auto_detect_and_open` logic.
- Deliberately **not** wired into `mesh.configure`/the frontend — dev/verification-only surface
for now (hand-edit `mesh-config.json`), consistent with how narrowly scoped this phase is.
- `cargo check -p archipelago` + `cargo test -p archipelago` (mesh module): **108 passed, 0
failed, 1 ignored** (the pre-existing hardware-gated `probe_rnode_detects_real_hardware`) —
zero regression to the serial/RNode path, provable without any hardware.
4. **End-to-end Rust integration test** (`mesh::tests::mesh_service_connects_over_reticulum_tcp_client`,
`#[ignore]`d — spawns real subprocesses, skipped in the default `cargo test` run the same way
the rest of the mesh suite skips hardware-gated tests): a real `MeshService::start()` spawns the
daemon in TCP **client** mode (no serial probe at all), dials a second stand-alone daemon
instance in TCP **server** mode (the Aurora-side role), and reaches `device_connected: true` /
`device_type: Reticulum` via the exact `MeshService::status()` call the `mesh.status` RPC uses.
Passed in ~2.6s. Run manually: `cargo test -p archipelago -- --ignored
mesh_service_connects_over_reticulum_tcp` (needs `reticulum-daemon/.venv`, see below).
**Environment note:** this session's Rust toolchain drift — system `rustc` (apt, 1.85.0) is too
old for code already on `main` (`u32::is_multiple_of` in `health_monitor.rs`, stabilized upstream
after 1.85); a pre-installed rustup toolchain at
`~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu` (1.96.0) builds clean. Not something this
phase's changes caused — pre-existing, just newly hit. Put that toolchain's `bin/` first on `PATH`
if `cargo check`/`test` reports `E0658 unsigned_is_multiple_of`.
**Explicitly NOT done (out of scope for this phase, see plan non-goals):**
- Real Aurora Flutter GUI verification — this dev sandbox has no `flutter`, no `$DISPLAY`, and no
`reticulum-dart` sibling checked out (Aurora's actual RNS implementation lives in that separate
repo; Aurora's CI clones it fresh at build time). The scripted-stand-in gate above is the
protocol-level substitute. **Manual follow-up**: point a real Aurora build's TCP hub list (or an
ad hoc connect) at an archy node's `--tcp-listen` address and confirm an LXMF DM in the actual
app UI.
- Any non-loopback (LAN/WAN) TCP bind — hard-gated off on purpose; a real "Aurora hub" deployment
needs its own security review given archy's Tor-first posture for inter-node traffic.
- LXMF propagation-node / always-on-hub role for archy (bridging Aurora's offline BLE peers) —
bigger architectural + storage commitment.
- Identity unification between archy's and Aurora's independent Nostr/secp256k1 keys — both
already have separate Nostr identities with no derivation link; out of scope here.
- `mesh.configure` RPC / frontend exposure of `reticulum_tcp` — stays hand-edit-only until/unless
it becomes user-facing.
docs/ROADMAP.md
+3 -5
View File
@@ -1,8 +1,7 @@
# Archipelago Roadmap
_Last updated: 2026-07-08. This is the public-facing summary. The live,
priority-ordered engineering list is [`UNIFIED-TASK-TRACKER.md`](UNIFIED-TASK-TRACKER.md);
the narrative plan behind it is [`PRODUCTION-MASTER-PLAN.md`](PRODUCTION-MASTER-PLAN.md)._
open engineering work is tracked in the public issue tracker._
## North star
@@ -50,9 +49,8 @@ security-first.
## 🔄 In progress
- **Multinode pass** — run the same production gate across the whole test
fleet, plus cross-node federation/mesh suites
([`multinode-testing-plan.md`](multinode-testing-plan.md)). This is the
current exit criterion.
fleet, plus cross-node federation/mesh suites. This is the current exit
criterion.
- **Quadlet default flip** — flip the validated Quadlet path from opt-in to
default fleet-wide; eliminates the last container-flapping root cause.
- **Container-flapping elimination** — reconciler churn and failed-unit
docs/SEED-VERIFICATION.md
+25 -6
View File
@@ -19,6 +19,9 @@ PBKDF2-HMAC-SHA512 (2048 rounds, salt = "mnemonic")
+-- HKDF-SHA256 (info="archipelago/nostr-node/secp256k1/v1")
| --> Node Nostr key --> npub1...
|
+-- HKDF-SHA256 (info="archipelago/fips/secp256k1/v1")
| --> FIPS mesh transport key --> npub1...
|
+-- HKDF-SHA256 (info="archipelago/identity/{i}/ed25519/v1")
| --> Identity[i] Ed25519 --> did:key:z...
|
@@ -265,6 +268,20 @@ def main():
print(f" nsec: {bech32_encode('nsec', node_nostr_priv)}")
print(f" npub: {bech32_encode('npub', node_nostr_pub)}")
# ── 2b. FIPS mesh transport key ─────────────────────────────────────
print()
print("-" * W)
print(" 2b. FIPS MESH TRANSPORT KEY")
print(f" HKDF-SHA256(seed, info='archipelago/fips/secp256k1/v1')")
print("-" * W)
fips_priv = hkdf_sha256(seed, b"archipelago/fips/secp256k1/v1")
fips_pub = secp256k1_xonly(fips_priv)
print(f" X-only: {fips_pub.hex()}")
print(f" npub: {bech32_encode('npub', fips_pub)}")
# ── 3. Identity[0..2] Ed25519 + DID ─────────────────────────────────
print()
@@ -324,10 +341,12 @@ def main():
print()
print("=" * W)
print(" Compare these values with your Archipelago node:")
print(" UI: Settings > Identity")
print(" SSH: xxd -p /var/lib/archipelago/identity/node_key.pub")
print(" RPC: curl -s http://<ip>/api/rpc \\")
print(" -d '{\"method\":\"identity.get-node\"}' | jq .")
print(" SSH: xxd -p /var/lib/archipelago/identity/node_key.pub (section 1)")
print(" cat /var/lib/archipelago/identity/nostr_pubkey (section 2)")
print(" RPC: curl -s -b jar.txt http://<ip>/rpc/v1 \\")
print(" -H 'Content-Type: application/json' \\")
print(" -d '{\"method\":\"node.did\"}' | jq .")
print(" ...and {\"method\":\"node.nostr-pubkey\"} for the npub")
print("=" * W)
print()
@@ -371,12 +390,12 @@ RPC alternative (from any machine on the LAN):
```bash
# Node identity
curl -s http://192.168.1.228/api/rpc \
curl -s http://archipelago.local/api/rpc \
-H 'Content-Type: application/json' \
-d '{"method":"identity.get-node"}' | jq .
# All identities
curl -s http://192.168.1.228/api/rpc \
curl -s http://archipelago.local/api/rpc \
-H 'Content-Type: application/json' \
-d '{"method":"identity.list"}' | jq .
```
docs/UNIFIED-TASK-TRACKER.md
-435
View File
@@ -1,435 +0,0 @@
# Unified Task Tracker — OTA 1.8.0 + Master Plan
Single working list for everything left before 1.8.0 ships and the next master-plan
exit criteria (multinode + workstreams B/C/D) are met. Supersedes the open-task
sections of `docs/archive/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md`
as the day-to-day tracker — those docs remain the historical record / detailed
narrative and are still linked from here where useful. **Ordered fastest/simplest
first** so we work top-down instead of hunting across docs.
Verified against actual code state on 2026-07-01 (not just doc text — several
items the source docs still listed as "open" turned out to already be shipped;
those are marked ✅ below with the commit that did it, so we stop re-litigating them).
---
## Tier 0 — Quick / mechanical, no blockers
- [ ] **Ship the lightning payment false-failure fix in the next release** (fixed
on main 2026-07-27, needs OTA). Slow multi-hop payments (>15s) surfaced as
"Payment failed" while LND settled them in the background — the shared LND
REST client's 15s timeout aborted the synchronous `/v1/channels/transactions`
wait. Now: payinvoice decodes the invoice first for its payment hash, waits
up to 120s on a dedicated client, returns `status: "pending"` (never a
failure) on timeout, and the new `lnd.paymentstatus` RPC + frontend
`payLightningInvoice()` helper poll to a real terminal state (all 5 UI call
sites migrated). Verify on Framework PT with a real multi-hop payment.
- [ ] **Show the app version on the companion mobile-app banner in the app store
and on its install/pairing modal** (user request 2026-07-27) — so it's
obvious at a glance whether the node is serving the latest APK build.
- [ ] **Optimise the companion QR scan — quicker + better** (user request
2026-07-27; deferred to a later session on purpose). The pairing/scan QR
flow works (user-verified on-device 2026-07-27) but should get faster and
smoother: quicker camera start + decode (scan resolution/framerate,
continuous autofocus), more forgiving in low light / at an angle, and
snappier feedback once the code locks. Touch the native-scan path from
PR #104 and the in-app scan modal together so both benefit.
- [ ] **Update `tests/lifecycle/TESTING.md`'s stale Release Gates checklist** (lines
289–296) — several boxes are unchecked but actually true now:
- #1 bitcoin-stops: covered by `tests/lifecycle/bats/bitcoin-knots.bats` stop/restart
tier, included in the 5/5 green gate run.
- #2 `ARCHY_ITERATIONS=5` on .228: **GREEN 2026-06-23 per CLAUDE.md** — check the box.
- #5 cargo 0 warnings: confirmed 0 warnings on `cargo build --release` (2026-07-01).
- #7 layman changelog: `CHANGELOG.md` is backfilled with layman-readable entries
through v1.8.00-alpha — check the box.
- Leave #3 (multinode), #4 (backend-survives-restart / Phase-3 default-on), #6
(LoC decision), #8 (tag pushed) unchecked — genuinely still open, see Tier 2/3.
- [x] ~~Finish the archival/full-node manifest generalization~~ — investigated 2026-07-01:
the hardcoded fallback names in `dependencies.rs:48-52` (`electrs`, `mempool-electrs`,
`mempool-web`) are legacy **alias** ids for `electrumx`/`mempool`, resolved via
id-mapping in a dozen other places (`install.rs`, `runtime.rs`, `config.rs`, etc.),
not separate un-migrated apps with their own manifests. `electrumx` and `mempool`
themselves already declare `bitcoin:archival`. The fallback is correct as-is —
not tech debt, closing this item rather than risk breaking alias resolution.
- [x] ~~Confirm/close the Portainer image-pin item~~ — confirmed 2026-07-01:
`146.59.87.168:3000/lfg2025/portainer:2.19.4` is present in `podman images` on
all 3 LAN nodes (.116/.198/.228), i.e. actually resolvable/pulled from the mirror.
Not a live bug.
- [x] ~~grafana Quadlet "stuck activating"~~ — checked live on .116 (2026-07-01):
`grafana.service` is `active (running)`, container `Up 2 hours (healthy)`. The
2026-06-21 report is stale for grafana. **strfry still unconfirmed** — not
installed on any of .116/.198/.228 to check directly; low priority until someone
actually needs it installed.
- [ ] **Add `cargo audit` / `cargo deny` to CI, failing on duplicate `rand` majors**
(entropy audit R-05, finding F-07 —
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). `cargo-audit` is not installed
anywhere, so no RustSec check has ever run against this tree. Separately,
`cargo tree` shows **both** `rand 0.8.5` (direct, all first-party key generation)
and `rand 0.9.2` (transitive via `totp-rs` and `tungstenite 0.26.2`) resolved into
one binary. `rand 0.9.0` removed `ThreadRng` fork protection and the orchestrator
forks constantly, so a future bump must be visible rather than silent — add a
`bans` rule so the duplicate majors show up in CI, not in an incident.
- [ ] **Harden the release signing ceremony's mnemonic input** (entropy audit R-08,
finding F-06). `ceremony gen` prints the release master mnemonic to **stdout**
(`core/archipelago/src/ceremony.rs:71-77`) and `load_release_root_key` prefers the
`RELEASE_MASTER_MNEMONIC` **environment variable** over stdin (`:157-160`) — both
leak into shell history, `/proc/<pid>/environ`, tmux scrollback and terminal
recordings. This is the seed that derives the fleet release-root signing key, so a
leak means forged signed manifests fleet-wide. Make stdin/TTY the only supported
input for `sign`/`pubkey`; write `gen`'s output to a `0600` file rather than the
terminal. Small change, but schedule it deliberately — it is the signing ceremony.
- [ ] **Small entropy-audit hygiene batch** (entropy audit R-09 – R-12, R-14 —
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). Five independent one-liners,
each closing a Low/Informational finding:
- Persist the CSPRNG-readiness verdict (`seed.rs:85-91`) as a durable structured
event, so any node can answer post-hoc "was the entropy pool ready when this seed
was born?" — the question Coldcard owners cannot answer today.
- Add a test asserting the `getrandom` crate uses the **blocking** syscall, making
`seed.rs:52-57`'s invariant mechanical instead of a comment.
- Clear `_seed_words` from `sessionStorage` on route-leave from onboarding, not only
on successful verify (`OnboardingSeedVerify.vue:251`), plus a wall-clock expiry
mirroring the server's 10-minute `MNEMONIC_TTL`.
- Replace `% charset.len()` in `totp.rs:305` with `SliceRandom::choose(&mut OsRng)`.
(No bias today — 32 divides 256 — but any future charset edit introduces one
silently. The audit refutes the research's claim that this is currently biased.)
- Comment `pickRandomIndices` (`OnboardingSeedVerify.vue:157`) to record that its
`Math.random()` picks a UX challenge, not key material, so the next auditor does
not re-derive that it is benign.
- [ ] ~~**Swap container `generated_secrets` to explicit `OsRng`** (entropy audit R-13,
finding F-10) — two-line change in `container/secrets.rs:90-102`~~
**SUPERSEDED 2026-08-02 by R-16 / KEY-05.** The audit scoped this at 2 call sites; the
real surface is **41 across 15 files** — see the audit's new §F-10a. `secrets.rs` is 2
of them, and a two-line fix there while 39 other sites inherit the same dependency
default is not a fix.
- [ ] **Crate-wide CSPRNG enforcement — a defaulted RNG cannot be inherited anywhere**
(entropy audit **R-16 / F-10a**, Medium) — tracked as **KEY-05 in Phase 10**, so plan
and execute it there rather than as a standalone item. `session.rs` (16 sites),
`pine_ha.rs` (6), `wallet/bdhke.rs` (2 prod — **Cashu proof secret + blinding factor,
genuine key material**), `storage_crypto.rs` (1 — **AEAD nonce**), `mesh/x3dh.rs` (2 —
prekey *identifiers*, **not** key material — corrected 2026-08-02), +10 more files.
Nothing is broken today (`rand::random()`/`thread_rng()` are ChaCha12 from
`getrandom(2)`), but it is the T1 shape that produced the COLDCARD defect, now with key
material in the blast radius. Five layers: sealed allowlist trait at key-gen seams;
`clippy.toml` `disallowed-methods` ban (compile-time, CI-enforced — no `clippy.toml`
exists yet); `cargo-deny` on duplicate `rand` majors (absorbs R-05); degenerate-entropy
runtime check; persist the CSPRNG-readiness verdict (absorbs R-09). Also retires the
`impl rand::CryptoRng for CountingRng` false promise at `seed.rs:656`.
**Gated: do not start until the concurrent Phase 1 agent is done and synced.**
## Tier 1 — Medium effort, unblocked
- [ ] **Fix the fail-open first-boot secret regeneration in the ISO** (entropy audit
R-02 + R-03, finding F-03 — `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`).
The installed rootfs is a **cached container export shared by every node**
(`image-recipe/_archived/build-auto-installer-iso.sh:717-726`, extracted at
`:2303`), and it bakes SSH host keys (via the `openssh-server` install at `:345`)
and a TLS keypair (`:463-469`). `archipelago-first-boot-secrets.service` correctly
regenerates both per device — but both branches are **fail-open** (`:1647`,
`:1659`) and `touch "$MARKER"` at `:1663` runs **unconditionally**, so a single
transient failure permanently leaves that node on the image-wide shared SSH host
key and TLS private key, with the failure visible only in a log file. Fix:
(a) set the marker only when both regenerations succeeded, so it retries next
boot; (b) surface the failure in the UI/doctor, not just the log; (c) strip the
baked keys from the rootfs tar so a failure degrades to "no key" rather than
"shared key". Needs an ISO rebuild and two fresh flashes to verify.
- [ ] **Reconcile `Argon2::default()` with ADR-005** (entropy audit R-06, finding F-05).
ADR-005 states 64 MB / 3 iterations
(`docs/adr/005-chacha20-backup-encryption.md:31`); `Argon2::default()` in
argon2 0.5.3 is Argon2id at **19 MiB / t=2 / p=1**. Used at
`core/archipelago/src/seed.rs:249` and `:285`, `backup/identity.rs:38`/`:93`,
`backup/full.rs:618`/`:650`. Either raise the parameters behind a versioned
envelope **with a migration** (an existing `master_seed.enc` was encrypted under
the old parameters and will not decrypt under new ones) or amend the ADR to state
the real numbers. Do not change them silently.
- [ ] **Run the on-node entropy verification checklist** (entropy audit R-15, §6 of
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). Everything in that section is
explicitly **UNVERIFIED** — it needs real hardware this environment cannot reach.
Highest value first: **C-3** (are SSH host-key and TLS fingerprints actually
different across two nodes flashed from the same ISO?) and **C-5** (the cross-node
same-ISO seed collision test — the empirical check that would have caught the
Coldcard defect). Also C-1 (`crng init done` vs seed-generation timestamp), C-2
(`machine-id` uniqueness), C-4 (what the rootfs tar actually contains, run on the
build host), C-6 (is `/rpc/v1` reachable unauthenticated from the LAN). Use a
disposable node — C-5 overwrites node identity.
- [x] ~~immich → Quadlet migration~~ — investigated 2026-07-01, turned out already done:
immich uses the same `install_stack_via_orchestrator` primitive as netbird/btcpay
(`immich_stack_app_ids()` in `stacks.rs:690`), and is confirmed running as real
Quadlet units live on .228 (`immich_server.container`, `immich_postgres.container`,
`immich_redis.container`, all active). Not a legacy in-cgroup app — the only
remaining piece is the fleet-wide Phase-3 default-flip, already tracked in Tier 2.
- [x] ~~Netbird reinstall adoption path~~ — investigated 2026-07-01, **not a bug, by
design.** `adopt_stack_if_exists()` (`stacks.rs:140-198`) is only used as a
fallback when the orchestrator has no manifest for the app — there's nothing to
render certs/config from in that case, so skipping rendering is correct. When
the orchestrator *does* have the manifest (the normal path), the reconcile loop
already re-renders certs even for adopted-running containers, fixed in
`4519dbf0` (`prod_orchestrator.rs:1707-1708`).
- [x] ~~TanStack Query (or equivalent) investigation~~ — spike complete 2026-07-01,
**recommendation: don't adopt / close as not needed.** Only 3 stores actually fetch
data, WebSocket push already handles hot data (server-info/package-data), no
cache-invalidation or stale-data bugs found, migration would touch 62 RPC call
sites for no concrete payoff. If boilerplate ever bothers us, extract a
`usePolling()` composable instead — much cheaper than a query-cache migration.
## Tier 2 — High effort, mostly unblocked (the actual next exit criteria)
- [ ] **🔴 Gate the unauthenticated seed RPCs** (entropy audit R-01, finding **F-01,
Critical** — `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`). `seed.generate`,
`seed.verify`, `seed.restore` and `seed.save-encrypted` are in
`UNAUTHENTICATED_METHODS` (`core/archipelago/src/api/rpc/middleware.rs:24-28`),
which skips session, RBAC **and** CSRF (`api/rpc/mod.rs:263`, `:295`, `:326`).
Neither handler checks whether onboarding is already complete
(`api/rpc/seed_rpc.rs:93-159`, `:226-305`), and `NodeIdentity::from_seed`
overwrites `node_key`, `nostr_secret` and the FIPS mesh key **unconditionally**
(`identity.rs:79-114`). There is no rate limit (`rate_limit.rs:60-97` has no
`seed.*` entry). The endpoint is proxied to the LAN over plaintext HTTP
(`image-recipe/configs/nginx-archipelago.conf:11`, `:165`, `:192`) and mesh peers
can reach it too (`server.rs:2080` asserts `/rpc/v1` passes the peer path filter).
Net: **one unauthenticated POST can take over or destroy a live node's identity**,
and `seed.restore` lets the attacker choose the mnemonic. The guard already exists
and is simply never called — `NodeIdentity::key_exists` (`identity.rs:117`).
Fix: bail when a node key exists and no onboarding mnemonic is pending; prefer
also gating on `auth_manager.is_onboarding_complete()`; add rate limits at
`auth.changePassword` strictness; narrow the peer path filter. Changes an
authentication boundary on a live fleet — **needs its own `/gsd-plan-phase` with a
federation re-verify**, not an opportunistic patch.
- [x] **PSBT-first signing: Phase 1 — move the Bitcoin private key out of Core** — **DONE
2026-08-02 by deletion, not conversion** (entropy audit R-04, finding **F-13**;
Phase 10 plan 10-05, decision **D-07b**). The handler that imported the BIP-84
account **private** key into Core's `wallet.dat` had no caller anywhere, LND is the
wallet the UI drives, and the endpoint was authenticated *and* password-gated — so
it was deleted outright rather than rewritten watch-only. `bitcoin.rs`'s wallet-init
handler and its `dispatcher.rs` arm are gone; **no daemon code path writes the
BIP-84 private key into Bitcoin Core.** No migration was performed or is needed —
a 4-node fleet census found no wallet the handler created. D-09's key-origin
requirement moved to the PSBT itself: `lnd.create-psbt` now reports
`key_origin` (`psbt_key_origin_report`, `api/rpc/lnd/wallet.rs`).
**Read `docs/security/KEY-03-SIGNING-POSTURE.md` for the current state** — it also
records the verdict that **no fleet node is provisioned watch-only**, so what ships
today is PSBT *transport*, not air-gapped custody.
- [ ] **Finish the Core-wallet fleet census — 6 nodes unchecked** (Phase 10 plan 10-05,
Task 3; standing item). The 2026-08-02 census examined 4 nodes (archi-dev-box,
shorty-s/.228, archy-x250-beta, archy-x250-pa) and found **no** wallet created by
the deleted handler and no wallet holding keys or funds. Six were not examined:
framework-pt, archipelago-1, archipelago, archy-dev-pa and archipelago-5
(SSH auth/connectivity) and archy-x250-dev (offline). Re-run the **read-only**
procedure in `docs/security/KEY-03-SIGNING-POSTURE.md` § *Fleet census* when
credentials or connectivity allow — a natural fold-in for KEY-04's on-node work.
**Never run `listdescriptors true`** (it returns private keys). If any node reports
a wallet named `archipelago`, or any descriptor wallet with
`private_keys_enabled: true` that is not blank/empty, **stop and escalate — do not
migrate or modify it** (D-07b).
- [ ] **PSBT-first signing: Phases 2-7 rollout**
(`docs/security/PSBT-SIGNING-ARCHITECTURE.md` §8) — the spec is written to be
consumed directly by `/gsd-plan-phase`, with per-phase goals, dependencies,
candidate requirements and hardware gating. Sequence: PSBT construct/export →
external-signer import + finalize → air-gap transport (BC-UR v2 primary, BBQr for
Coldcard, file fallback always) → `wsh(sortedmulti)` multisig on BIP-48 → LND
remote signing → hot-wallet spend limits and cold/warm/hot tiering. Two hard rules
the spec fixes in place: a channel-funding PSBT must **never** be self-broadcast
(funds can be lost), and no UI copy may imply a routing node's Lightning channel
keys are cold — they are necessarily hot. Phases 3-6 need real hardware.
- [ ] **Confine the seed-bearing RPCs to loopback/TLS** (entropy audit R-07, finding
F-04 / [ARCHY-4]). The 24-word master mnemonic is returned to the browser over
JSON-RPC (`core/archipelago/src/api/rpc/seed_rpc.rs:147`, `:156-158`), held in
process memory under a 10-minute TTL (`:27`) and deliberately **not** cleared at
verify time (`:205-211`, with a documented and defensible rationale about client
retries) — over a transport that is plaintext HTTP on LAN by design
(`api/rpc/mod.rs:227-241`). Anyone with LAN traffic visibility during onboarding
reads the phrase that unlocks the wallet and the node identity. Fix: force TLS or
loopback for seed methods, shrink the TTL, and clear on an acknowledged verify
with a short grace window. Touches the onboarding transport — needs a phase.
- [~] **Multinode test pass** (`docs/multinode-testing-plan.md`) — worked the
preconditions on .198 2026-07-01:
- ✅ cleared 2 stale failed-unit records (`archy-mempool-db.service`,
`meshtastic.service` — both `not-found`/dead since 6 and 5 days ago, harmless
bookkeeping, `systemctl --user reset-failed`).
- ✅ nginx `/app/lnd/` proxy target confirmed correct (→ `18083`, matches the
running `archy-lnd-ui` port) — the plan's "stale proxy target" concern doesn't
apply here.
- ⛔ .198 disk (448GB) is below the 1TB archival threshold + was only 21%
through IBD — user chose to **swap in a different node** rather than wait/add
storage. **.116 ruled out** (no bitcoin container installed at all, just the
UI companion). **.120 ruled out** (reserved for another developer). **.5**
(archy-x250-beta, Tailscale `100.72.136.5`) chosen: also sub-1TB (472GB, so
still pruned — that ceiling is shared by every non-.228 node), but **fully
synced** (`ibd:false`, blocks==headers 956,240). Bootstrapped bats 1.11.1 +
jq 1.7.1 onto it 2026-07-01 and **launched the 5× destructive gate
(`ARCHY_ITERATIONS=5 ARCHY_ALLOW_DESTRUCTIVE=1`) — running now**, log at
`/tmp/gate.log` on .5, background poller watching for the `RESULTS` banner.
- Once .5's gate reports: bring the rest of the fleet to precondition, then the
cross-node federation/mesh/transport suites. This is the literal
"next exit criterion" called out in `CLAUDE.md`.
- [ ] **Phase-3 Quadlet default-flip** — code is validated + opt-in via
`ARCHIPELAGO_USE_QUADLET_BACKENDS=true` on .228/.198 already (confirmed live
2026-07-01). Ready to flip (`config.rs:256` + its test) the moment the .5 gate
reports clean — deliberately NOT staged uncommitted in the tree (a prior attempt
left an uncommitted flip sitting around and that caused confusion; it's a 2-line
change, faster to just do it fresh once confirmed).
- [x] ~~Per-app test coverage for the ~30 apps with zero automated coverage~~ —
**reframed 2026-07-01, mostly a non-issue.** `all-apps-matrix.bats` +
`all-apps-lifecycle.bats` already give EVERY installed app generic baseline
coverage (no stuck state, no error state, stop/start/restart survives, UI
reachable). The real gap is narrower: **34 apps lack app-specific assertions**
(health endpoints, API queryability, data integrity) beyond that baseline —
aiui, bitcoin-core, botfights, core-lightning, did-wallet, fedimint-clientd,
fedimint-gateway, fips-ui, gitea, grafana, home-assistant, indeedhub (+5
sub-containers), jellyfin, lightning-stack, lnd-ui, morphos-server, netbird
(+2 sub-containers), nextcloud, nostr-rs-relay, photoprism, portainer, router,
searxng, strfry, uptime-kuma, vaultwarden. Not urgent — baseline coverage is
real safety net; treat as a backlog "nice to harden further," not a gate item.
- [x] ~~Convert remaining multi-container legacy stacks to the manifest-owned model~~ —
**investigated 2026-07-01, DONE, nothing left.** All 5 real multi-container
stacks (btcpay, mempool, immich, netbird, indeedhub) are on the
`install_stack_via_orchestrator` pattern (`stacks.rs`). saleor was removed from
the codebase; portainer/home-assistant/grafana are single-container
manifest-driven apps, never stacks; fedimint/fedimint-gateway/fedimint-clientd
are 3 separate single-container apps with manifest dependency edges, not a
coordinated stack. Workstream A's stack-migration tail is fully closed.
- [ ] **Container thrashing/flapping + reconciler churn** (added 2026-07-04 — was
implicit across other tracks, now an explicit pre-tag concern). The root cause
of restart-storm flapping is pre-Quadlet architecture: restarting
`archipelago.service` SIGKILLs every container in its cgroup, then the
reconciler rebuilds the world over several minutes (the post-OTA health check
deliberately skips per-app container assertions because of exactly this).
Consolidated lever list, in order of impact:
- **Phase-3 Quadlet default-flip** (tracked above) — removes the SIGKILL-the-world
behavior entirely; the single biggest fix.
- **Workstream F lifecycle items** — immich/grafana uninstall hangs + ghost
containers, grafana reinstall stops, fedimint guardian sync
(`docs/PRODUCTION-MASTER-PLAN.md` workstream F).
- **Reconciler churn observability** — no metric/log today distinguishes "settling
after restart" from "flapping"; add a per-app restart counter + log line when an
app restarts >N times in M minutes so thrash is visible instead of anecdotal.
- **Failed-unit self-healing gap (observed live 2026-07-06 on .228)**: fedimint's
quadlet unit exited 255 at 21:21 and sat `failed` for 7+ hours — the reconciler
never revived it (it repairs missing/drifted containers but doesn't
`reset-failed`+start failed .services). Same for the indeedhub trio after the
gate run. The health monitor also can't help (container is gone when the unit
fails). Add a reconcile step: quadlet-backed app whose .service is `failed` and
not user-stopped → reset-failed + start, with backoff.
- Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01),
indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`,
removes executor stalls that made the API janky under reconcile load),
quadlet entrypoint-split false-drift fix (2026-07-08 — `container_command_drifted`
compared entrypoint/cmd halves separately, but quadlet folds `sh -lc` into
`Entrypoint=sh` + `Exec=-lc …`, so every quadlet-created app with a
multi-element entrypoint read as permanently drifted; electrumx on .228
recreated 114×/6h until the comparator was switched to concatenated argv).
- Perf polish riding along: 93 MB frontend dist shrink (hardening plan §D 🟡).
- [ ] **Developer tooling CLI suite** (validate/render/local-install/lifecycle-test) —
APP-PACKAGING-MIGRATION-PLAN.md step 5, needed before external devs can publish.
- [x] ~~**Consolidated deploy 2026-07-01**: merged PR #67 (reticulum daemon
process-group fix, `469b0203`), the UI/UX work (`8256fde1` — mesh/web5/apps
layout, modal, search UX), and `archy-openwrt` (TollGate/OpenWrt gateway
integration — new `core/openwrt` crate, RPC surface, `OpenWrtGateway.vue`)
into `main`, alongside the indeedhub self-heal fix~~ — all merged clean, no
conflicts. **Found + fixed 2 real build-breaking issues during
verification, not caught by whoever authored them**: a vestigial unused
`ref` in `Web5ConnectedNodes.vue` that broke `vue-tsc`, and a stale
`MeshMap.test.ts` mock missing `federatedPositions` (predated this
session's Mesh Map feature) that crashed on mount. Full test suite green
(667 passed) after fixes. **Deployed fleet-wide 2026-07-01, all 5 nodes
sha256-verified**: .116, .198, .228, .5 (recovered cleanly from one
truncated-transfer hiccup, caught via checksum before it hit the live
service), 100.82.34.38 (non-Quadlet node — all containers survived the
restart intact, unlike the worst-case risk flagged beforehand). Also
built an unbundled installer ISO from this same merged source
(`archipelago-installer-1.7.99-alpha-unbundled-x86_64.iso`, 2.4GB) —
the ISO pipeline was archived from the release process at v1.7.43-alpha
(OTA tarballs are now primary) but the wrapper script still works.
- [ ] **⚠️ NOT YET DEPLOYED — start here next session.** After the fleet deploy
above, found that PR #67 ("kill whole daemon process group on drop",
branch `fix/reticulum-daemon-process-group`, head `be50c886`) is a
**different, separate** reticulum-daemon fix from the one already
deployed (`469b0203` on `fix/reticulum-daemon-pdeathsig`) — I'd
conflated the two by topic similarity and only merged/deployed the
Python-level `pdeathsig` fix, missing PR #67's Rust-level
kill-whole-process-group-on-`Drop` fix entirely. Merged PR #67 into
`main` (`7a7fec21`, clean, `cargo check` green, complementary not
conflicting with the already-deployed fix) and separately fixed a real
bug found live: `OpenWrtGateway.vue`'s back button had no `@click`
handler at all (`7d7ba573`, `vue-tsc` clean). **Both committed + pushed
to `main` but genuinely NOT deployed to any node** — user asked to hold
off deploying to restart their computer. Also spot-checked
`openwrt.scan` live on .116: RPC plumbing works, but no physical
OpenWrt router was available to confirm true-positive detection, and
`detect::scan_subnet` does blocking TCP/SSH calls inside an `async fn`
with no `.await` — untested at scale, worth hardening. **Next steps**:
build release binary + frontend from current `main`, deploy to all 5
fleet nodes (.116/.198/.228/.5/100.82.34.38) the same way as the
earlier consolidated deploy, then verify the back button + (if a real
OpenWrt router is available) router detection live.
- [~] **Cross-node federation/mesh/transport suites** — **big find 2026-07-01: these
already exist**, just aren't wired into the gate or documented as existing:
`tests/multinode/smoke.sh` (federation pairing/sync, FIPS anchor, peer content
browse, tombstone-removal regression tests), `tests/multinode/meshtastic.sh`
(8-stage on-air mesh test), harness in `tests/multinode/lib/multinode.bash`.
**Actually ran `smoke.sh` live against .116↔.228 2026-07-01: 14 passed, 1
failed, 1 skipped.** Confirms federation pairing (both directions), FIPS
anchor connectivity (both nodes), and peer-content-browse-over-mesh (the
v1.7.95 fix) all genuinely work node-to-node right now.
- ⚠️ **Real robustness gap found**: `node_rpc()` in `tests/multinode/lib/multinode.bash`
has no `--max-time` on its curl calls — a slow server-side RPC hangs the whole
suite with zero feedback (this is what looked like a hang before it eventually
completed on its own). Cheap fix, not yet applied.
- 🐛 **Real regression found and root-caused**: removing a federation node
(`federation.remove-node`) doesn't reliably stick — B reappeared in A's peer
list after removal in the live test. Root cause: `remove_node()`
(`core/archipelago/src/federation/storage.rs:187`) does
`let _ = tombstone_did(data_dir, did).await` — **silently swallows the
tombstone write's errors.** If that write fails (disk I/O, permission,
transient issue), the peer is removed from `nodes.json` but never actually
tombstoned, so the next background sync/notify-join re-adds it — the
tombstone check at `handlers.rs:592-599` passes because the DID was never
recorded as removed. Diagnosed as a **pre-existing logic gap**, not a fresh
regression from the v1.7.95 fix. **Not fixed yet** — this is federation/trust
code, deliberately not touching it blind; needs a careful fix (surface the
tombstone-write failure instead of swallowing it, and/or retry) plus
re-verification with `smoke.sh` before considering it closed.
## Tier 3 — Blocked on a decision or resource only you can supply
- [x] ~~Version naming decision~~ — **decided 2026-07-08: `1.8.0-alpha`.** Remaining
work is the mechanical bump + tag + push once the pre-tag items above close.
- [x] ~~Workstream B signing ceremony~~ — **done 2026-07-02.** `anchor.rs` pins
`RELEASE_ROOT_PUBKEY_HEX = 5d15cbee…9951` (signer
`did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur`); mnemonic held
offline per `docs/workstream-b-signing-runbook.md`.
- [ ] **Bitcoin multi-version fleet-wide OTA** — `.228` fully working on branch,
per your prior gating this rollout is explicitly held for your decision on
timing (`docs/bitcoin-version-bulletproof-rollout.md`).
- [ ] **3ccc stock-Meshtastic RF validation** — needs a live send/receive test with
physical radios in your hands; code fix is in place, just unverified live.
## Backlog — deferred, no scope decided, low priority
- [ ] **Marketplace protocol (workstream C)** — design-only (`docs/marketplace-protocol.md`),
no tooling/trust UX built. Future work, not urgent.
- [ ] **DHT distribution (workstream D)** — confirmed design-only, no code
(`docs/dht-distribution-design.md` explicitly says "Status: Design (no code yet)");
an experimental iroh provider skeleton exists behind a feature flag for future
PoC measurement, nothing fleet-facing.
- [ ] **Custom live voice-call protocol** — deprioritized 2026-07-01 per user request;
scope not yet decided. Revisit after the tiers above are worked down.
---
*Historical narrative and detailed per-session logs remain in
`docs/archive/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md` §6/§8b —
this doc is the live "what's left, in priority order" list. Update it (don't just
append to the old docs) as items close or new ones surface.*
docs/adr/004-tor-for-peer-communication.md
+26 -1
View File
@@ -1,6 +1,7 @@
# ADR-004: Tor Hidden Services for Peer Communication
**Status**: Accepted
**Status**: Accepted (2026-03) — **partially superseded in practice, see
Amendment below**
**Date**: 2026-03
## Context
@@ -33,3 +34,27 @@ Use Tor hidden services (.onion addresses) for all inter-node communication.
- Implement retry with backoff for Tor connections
- Container `archy-tor` runs automatically with host networking for hidden service access
- Federation sync interval (5 min) tolerates occasional connection failures
## Amendment (recorded 2026-08)
Two things in this ADR no longer describe the system. Both changes happened
without their own ADR, which is itself worth noting.
**1. Tor is no longer used for *all* inter-node communication — it is the last
fallback.** The transport layer now tries, in order, mesh radio → LAN → FIPS
overlay → Tor (`transport::TransportKind`, priority 1–4). The latency and
bandwidth costs listed above are exactly why: FIPS was introduced to carry WAN
peering that Tor made too slow, and direct LAN peering skips the overlay
entirely for co-located nodes. Tor's NAT-traversal and IP-privacy properties are
still what make it the dependable floor when the others are unavailable.
**2. Tor does not run as the `archy-tor` container.** It is the host's Debian
`tor` package, running as `debian-tor` and driven by the
`archipelago-tor-helper` path unit (`scripts/tor-helper.sh`), which installs a
staged `/etc/tor/torrc` and restarts the service. The migration was deliberate
and is still enforced: `scripts/container-doctor.sh` removes an `archy-tor`
container if it finds one and switches the node to system Tor. There is no
`apps/tor` manifest.
The decision to use onion services for peer reachability stands; only its
exclusivity and its packaging changed.
docs/adr/009-manifest-container-security.md
+22
View File
@@ -70,6 +70,28 @@ Some apps legitimately need elevated privileges:
- Build-time validation catches issues before deployment
- Override mechanism for legitimate exceptions (with audit trail)
## Implementation status
The decision above stands; this section records how much of it is actually
enforced today, because the "non-negotiable" table overstates it. Verified
against `core/container/src/manifest.rs` and `core/security/src/`:
| Constraint | Reality |
|---|---|
| `capabilities` drop-all + allow-list | ✅ **Enforced.** A capability outside the nine-entry allow-list is a parse error, so the app cannot install |
| Bind-mount confinement | ✅ **Enforced** (stronger than this ADR describes): sources must be under `/var/lib/archipelago/`, a named volume, or one of two reviewed exceptions |
| `readonly_root` / `no_new_privileges` | ◐ **Defaults, not gates.** Both default to `true` when omitted, but `validate_security()` does not reject an explicit `false` — the "reject manifests that violate mandatory defaults" step does not exist |
| `image_tag` pinned | ◐ **Preflight only.** `scripts/validate-app-manifest.sh` grades it; the parser accepts `:latest` and the app installs |
| `user` UID > 1000 | ❌ **Not validated.** The runtime manifest parser has no UID check at all (the marketplace schema has an advisory one, which is a different type) |
| `seccomp_profile` | ❌ **Does not exist.** The string `seccomp` appears nowhere in `core/` — not as code, not as a TODO |
| AppArmor | ❌ **Inert.** `container_policies.rs` can generate and `apparmor_parser -r` a profile, but its own comment says `TODO: Configure Podman to use the profile`. `security.apparmor_profile` is parsed into a manifest field that nothing ever reads |
So the accurate summary is: capability and mount confinement are hard gates,
the process-hardening flags are safe-by-default rather than enforced, and the
kernel-level sandboxing (seccomp/AppArmor) named in the decision was never
wired up. Closing the last two rows is tracked in
[`1.8.0-RELEASE-HARDENING-PLAN.md`](../1.8.0-RELEASE-HARDENING-PLAN.md).
## References
- `docs/app-manifest-spec.md` — Full manifest specification
docs/api-reference.md
+7 -6
View File
@@ -263,7 +263,7 @@ All endpoints use JSON-RPC over HTTP POST to `/rpc/v1`.
| Method | Params | Returns | Auth |
|--------|--------|---------|------|
| `mesh.status` | — | `{ enabled: bool, device: string?, nodes: MeshNode[] }` | Yes |
| `mesh.discover` | `{ timeout_secs?: number }` | `{ nodes: MeshNode[] }` | Yes |
| `mesh.peers` | — | `{ peers: MeshPeer[], count: number }` | Yes |
| `mesh.broadcast` | — | `{ ok: bool }` | Yes |
| `mesh.configure` | `{ enabled: bool, device?: string }` | `{ ok: bool }` | Yes |
@@ -381,18 +381,19 @@ All endpoints use JSON-RPC over HTTP POST to `/rpc/v1`.
## Example: cURL
```bash
# Login
curl -c cookies.txt -X POST http://192.168.1.228/rpc/v1 \
# Login (the password you created on the node's first-boot setup screen —
# there is no default password)
curl -c cookies.txt -X POST http://archipelago.local/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"auth.login","params":{"password":"password123"}}'
-d '{"method":"auth.login","params":{"password":"YOUR_NODE_PASSWORD"}}'
# Get system stats (authenticated)
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"system.stats"}'
# Get DID
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-H "Content-Type: application/json" \
-d '{"method":"node.did"}'
```
docs/app-developer-guide.md
+61 -14
View File
@@ -109,7 +109,7 @@ app:
| `app.container.pull_policy` | Pull behavior, usually `if-not-present` |
| `app.container.network` | Podman network setting such as `archy-net` or `pasta`; dangerous namespace-sharing modes are rejected |
| `app.container.entrypoint` / `custom_args` | Entrypoint and command override |
| `app.container.derived_env` | Environment values rendered from allowed host facts such as `HOST_IP`, `HOST_MDNS`, and `DISK_GB` |
| `app.container.derived_env` | Environment values rendered from host facts. The complete placeholder set is `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}`, `{{BITCOIN_HOST}}`; an unknown name or an unbalanced `{{` is a parse error, so typos fail loudly |
| `app.container.secret_env` | Environment values read from `/var/lib/archipelago/secrets/<secret_file>`, injected as podman secrets (never visible in `podman inspect` or unit files) |
| `app.container.generated_secrets` | Secrets the orchestrator creates on first use (`hex16`/`hex32`/`base64`/`bcrypt`) — self-healing, 0600, no host provisioning |
| `app.container.generated_certs` | Self-signed TLS certs materialised before create; CN/SANs rendered from host facts |
@@ -206,15 +206,28 @@ metadata:
## Security Requirements
These are enforced by the marketplace/catalog pipeline and the node. Non-compliant apps are flagged.
Two different things enforce these, and it's worth knowing which is which:
- **The Rust parser (`core/container/src/manifest.rs`)** is the hard gate. A
manifest that violates one of its rules fails to parse, so the app cannot be
installed at all.
- **`scripts/validate-app-manifest.sh`** is the submission preflight. It applies
the *policy* rules the parser doesn't encode, and grades them `fail`/`warn`.
### Mandatory
1. **No `:latest` tag** — Pin a specific version: `myapp:1.0.0`
2. **Read-only root filesystem** — `security.readonly_root: true` (use volumes for writable data)
3. **No privilege escalation** — `security.no_new_privileges: true`
4. **Minimal capabilities** — Drop all caps, only add required ones
5. **No host network unless explicitly approved** — keep `security.network_policy` isolated or bridge
1. **No `:latest` tag** — Pin a specific version: `myapp:1.0.0`. Checked by the
preflight script (a `fail` for new apps, a `warn` for existing manifests being
migrated), **not** by the parser — a `:latest` manifest still installs, so
pinning is on you.
2. **Read-only root filesystem** — `security.readonly_root: true` (use volumes
for writable data). This is the parser's default when you omit it.
3. **No privilege escalation** — `security.no_new_privileges: true`. Also the
parser's default when omitted.
4. **Minimal capabilities** — Drop all caps, only add required ones. The
allow-list below *is* parser-enforced: anything outside it is a parse error.
5. **No host network unless explicitly approved** — keep
`security.network_policy` isolated or bridge (`isolated` is the default).
### Allowed Capabilities
@@ -234,9 +247,16 @@ The parser currently accepts this allow-list. Keep capability requests minimal;
### Forbidden
- Namespace-sharing network modes such as `container:<name>` or `ns:<path>`
- Mounting system paths: `/`, `/etc`, `/var`, `/usr`, `/proc`, `/sys`
- `SYS_PTRACE`, privileged containers, Docker socket mounts, or rootful execution
- Hardcoded secrets in environment variables or images
- **Host bind mounts outside `/var/lib/archipelago/`.** This is an allow-list,
not a blocklist of "system paths": `volumes[].source` must be absolute and
start with `/var/lib/archipelago/`, or be a plain named volume (no slashes),
or be one of two reviewed exceptions (`/run/user/1000/podman/podman.sock`,
`/var/run/dbus`). `..` anywhere in the path is rejected. Everything else fails
to parse, so your app's data belongs under `/var/lib/archipelago/<app-id>`
- Capabilities outside the allow-list above — including `SYS_PTRACE`
- Privileged containers or rootful execution
- Hardcoded secrets in environment variables or images — use `secret_env` or
`generated_secrets`
## Container Best Practices
@@ -264,6 +284,15 @@ files:
Use `overwrite: false` for first-run defaults that users or the app may later modify. Use `overwrite: true` only for generated files the platform must own.
`files[].content` supports its own placeholder set — a different one from
`derived_env`:
| Placeholder | Renders to |
|---|---|
| `{{HOST_IP}}` / `{{HOST_MDNS}}` | Host facts (`hostname -I` / the node's `.local` name) |
| `{{NETWORK_GATEWAY}}` | The gateway of the app's Podman network, i.e. aardvark's DNS address. Use it as an nginx `resolver` so container names re-resolve per request instead of pinning a stale IP and 502-ing after a restart |
| `{{secret:NAME}}` | The trimmed contents of the `0600` secret `NAME` from the service-owned secrets dir. `NAME` must be a bare filename. Never logged |
### Health Checks
Define a health check endpoint in your container:
@@ -292,11 +321,14 @@ container:
network: archy-net
derived_env:
- key: BITCOIN_RPC_HOST
template: bitcoin-knots
template: "{{BITCOIN_HOST}}" # resolves to whichever Bitcoin app is installed
- key: BITCOIN_RPC_PORT
template: "8332"
```
Prefer `{{BITCOIN_HOST}}` over hardcoding `bitcoin-knots` — a node may be running
Bitcoin Core instead, and the placeholder resolves to whichever is present.
The `archy-net` Podman network provides DNS resolution between containers. Use `derived_env` for host facts like `HOST_MDNS` instead of hardcoding node-specific URLs.
## Catalog Generation
@@ -341,6 +373,21 @@ cmp -s app-catalog/catalog.json neode-ui/public/catalog.json
## Testing Your App
### Validate Your Manifest
Before anything else, check your manifest against the schema and the app-submission
rules:
```bash
./scripts/validate-app-manifest.sh apps/my-app/manifest.yml
```
It reports `STATUS: APPROVED` or `STATUS: REJECTED` with the specific failures —
including the ones that block submission, such as an unpinned `:latest` image
tag (new apps must pin a concrete version). It needs `python3` and PyYAML; it
tells you if either is missing. The Rust parser in `core/container/src/manifest.rs`
remains the canonical validator — this script is the fast local preflight.
### Local Testing
```bash
@@ -363,15 +410,15 @@ podman logs my-app
1. Install via the marketplace UI or RPC:
```bash
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-d '{"method":"package.install","params":{"id":"my-app","dockerImage":"docker.io/myorg/my-app:1.0.0"}}'
```
2. Verify the container is running:
```bash
curl -b cookies.txt -X POST http://192.168.1.228/rpc/v1 \
curl -b cookies.txt -X POST http://archipelago.local/rpc/v1 \
-d '{"method":"container-list"}'
```
3. Check the UI at `http://192.168.1.228/app/my-app/`
3. Check the UI. The app's detail page is `http://archipelago.local/dashboard/apps/my-app`; the embedded launch surface is `http://archipelago.local/dashboard/app-session/my-app`
### Validate Manifest
docs/app-manifest-spec.md
+24 -9
View File
@@ -6,14 +6,22 @@ document and the code disagree, the code wins. See
[`app-developer-guide.md`](app-developer-guide.md) for the authoring workflow._
Every app is a directory `apps/<id>/` containing a `manifest.yml` with a single
top-level `app:` block. Apps are purely declarative — the orchestrator owns the
entire lifecycle; there is no per-app installer code.
top-level `app:` block. Apps are declarative — the orchestrator owns the entire
lifecycle; there is no per-app installer code.
One honest caveat: seven first-party apps still get Rust-side pre-start work
through a hardcoded `match app_id` in `ProdOrchestrator::run_pre_start_hooks` —
`bitcoin-ui`, `filebrowser`, `lnd`, `archy-nbxplorer`, `btcpay-server`,
`fedimint-clientd` and `grafana` render or repair a config before start. That is
orchestrator code rather than a per-app installer, but it is not
manifest-declared, and the direction of travel is to replace each case with a
reusable manifest primitive.
## Top-level fields (`app:`)
| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `id` | string | ✅ | Lowercase alphanumeric + `-`/`_`. Must match the directory name. |
| `id` | string | ✅ | Lowercase ASCII letters, digits and single hyphens only — **no underscores**, no leading/trailing `-`, no `--` (`is_valid_app_id`). Should match the directory name, though nothing enforces that: the loader keys off `app.id`, so a mismatch silently registers the app under the id in the file rather than the folder. |
| `name` | string | ✅ | Display name. |
| `version` | string | ✅ | App version shown in the UI. |
| `description` | string | — | One-line description. |
@@ -45,7 +53,7 @@ Exactly **one** of `image` or `build` must be present (image XOR build).
| `network_aliases` | list of string | Extra DNS names on `network` (podman `--network-alias`) — lets stack members answer to short baked-in hostnames (`api`, `minio`, `relay`). |
| `entrypoint` | list of string | Entrypoint override. |
| `custom_args` | list of string | Extra positional args appended after the image. |
| `derived_env` | list | `- { key, template }` — template rendered against host facts at apply time. Allowed placeholders: `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}` (plus dependency-resolved facts such as the active bitcoin host). Never hard-code host specifics. |
| `derived_env` | list | `- { key, template }` — template rendered against host facts at apply time. The allow-list is exactly `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}`, `{{BITCOIN_HOST}}` (`DERIVED_PLACEHOLDERS`); an unknown name or unbalanced `{{` fails validation. `{{BITCOIN_HOST}}` resolves to whichever Bitcoin app is running (`bitcoin-knots` or `bitcoin-core`, defaulting to knots). Never hard-code host specifics. |
| `secret_env` | list | `- { key, secret_file }` — value read from `/var/lib/archipelago/secrets/<secret_file>` and injected as a **podman secret**, so it never appears in `podman inspect` or unit files. `secret_file` must be a bare filename (no `/`, no `..`). |
| `generated_secrets` | list | `- { name, kind }` — orchestrator materialises the secret on first use (0600, rootless service user, idempotent + self-healing). `kind ∈ hex16 | hex32 | base64 | bcrypt` (bcrypt writes `<name>` = hash and `<name>.pw` = plaintext). |
| `generated_certs` | list | `- { crt, key, common_name?, sans? }` — self-signed TLS materialised before create; CN/SANs rendered against host facts. |
@@ -104,11 +112,18 @@ hooks:
## Installation semantics
The orchestrator compiles the manifest into a rootless Podman **Quadlet unit
under `user.slice`** — the container survives backend restarts and reboots, and
a level-triggered reconciler converges drift every 30 seconds. Multi-container
apps are sets of per-member manifests installed together via the stack
orchestrator (`api/rpc/package/stacks.rs`) on an app-local network.
The orchestrator compiles the manifest into a rootless Podman container that
survives backend restarts and reboots, and a level-triggered reconciler
converges drift every 30 seconds (`BootReconciler::DEFAULT_INTERVAL`).
Multi-container apps are sets of per-member manifests installed together via
the stack orchestrator (`api/rpc/package/stacks.rs`) on an app-local network.
**Quadlet is not the default path.** `config.use_quadlet_backends` defaults to
`false`, so ordinary apps still take the legacy `podman create + start` path;
the Quadlet-unit-under-`user.slice` backend is opt-in per node (config key or
`ARCHIPELAGO_USE_QUADLET_BACKENDS`) and stays behind the flag until the
lifecycle harness has gone green against it. Companion UI containers are the
exception that do use Quadlet today.
## Distribution
docs/architecture.md
+7 -8
View File
@@ -29,8 +29,8 @@
│ Podman REST API socket + systemd Quadlet units
┌──────────────────────┴───────────────────────────────┐
│ ROOTLESS PODMAN CONTAINERS │
│ 50+ manifest-driven apps as user.slice Quadlet │
│ units — survive backend restarts, self-heal │
│ 50+ manifest-driven apps the orchestrator owns and │
│ self-heals; companion UIs run as systemd Quadlet │
└──────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────┐
@@ -104,9 +104,11 @@ never appear in `podman inspect` or unit files), `generated_secrets` /
`generated_certs` (self-healing), `network_aliases`, `data_uid`, and
allow-listed `post_install` hooks that run inside the app's own sandbox.
**Install** compiles the manifest to a rootless **Quadlet unit under
`user.slice`** — containers survive backend restarts and reboots.
Multi-container apps (BTCPay, Mempool, Immich, NetBird, IndeeHub) are sets of
**Install** creates a rootless container the orchestrator owns; the companion
UI containers run as systemd **Quadlet units under `user.slice`** (the
validated path being flipped to default for all apps), so those survive backend
restarts and reboots outright, and the reconciler rebuilds any container that
vanishes. Multi-container apps (BTCPay, Mempool, Immich, NetBird, IndeeHub) are sets of
per-member manifests installed via the stack orchestrator on an app-local
network with readiness gates and generated cross-service secrets. A
level-triggered **boot reconciler** converges actual state to desired state
@@ -225,9 +227,6 @@ mesh, steganography, store-and-forward outbox.
| [`app-manifest-spec.md`](app-manifest-spec.md) | The `manifest.yml` schema |
| [`user-walkthrough.md`](user-walkthrough.md) | End-user installation and usage guide |
| [`troubleshooting.md`](troubleshooting.md) | Diagnostic scenarios and solutions |
| [`operations-runbook.md`](operations-runbook.md) | Ops commands and emergency recovery |
| [`multi-node-architecture.md`](multi-node-architecture.md) | Federation protocol design |
| [`marketplace-protocol.md`](marketplace-protocol.md) | Decentralized app discovery via Nostr |
| [`PRODUCTION-MASTER-PLAN.md`](PRODUCTION-MASTER-PLAN.md) | North star and workstream narrative |
| [`UNIFIED-TASK-TRACKER.md`](UNIFIED-TASK-TRACKER.md) | Live, priority-ordered open items |
| [`archive/`](archive/) | Historical audits, session logs, shipped designs |
docs/archive/HANDOVER-2026-07-02-iso-feedback.md
-151
View File
@@ -1,151 +0,0 @@
# Handover — fresh-ISO feedback bug-bash (2026-07-02)
**For: the agent building the next ISO + fleet deploy.** All fixes below are
**merged and pushed: gitea-ai main = `f5d24796`** (merge of `c375ecc4`,
65 files; branch `iso-feedback-fixes-2026-07-02` also pushed). Source
feedback: user's fresh ISO install on a Framework (11th-gen Tiger Lake)
machine, node `192.168.1.81` (SSH `archipelago` / `archipelago`).
Diagnostic bundle: `/home/archipelago/incoming-logs/node-logs-192.168.1.81/`.
**⚠️ Known-red tests on main (NOT from this work):** `trust::anchor::
unset_constant_is_none` + 2 `trust::signed_doc` tests fail because a prior
commit pinned `RELEASE_ROOT_PUBKEY_HEX` without updating them. The signing/
audit agent's uncommitted changes in the shared tree fix exactly these —
coordinate with them; don't "fix" it independently or you'll collide. This
bug-bash branch alone was 898/898 green; merged with main it's 894/898 with
only those three.
## ⚠️ Outstanding user request for the deploy
- **Change .81's web-UI password to `ThisIsWeb54321@`** — the user forgot the
current one. Node was unreachable from .116 during this session (flaky WiFi
AP, IP flapped .68↔.81). Do this during deploy (SSH works from the user's
machine; `archipelago`/`archipelago`).
## What changed (by file)
### Backend (core/archipelago/src) — builds clean, targeted tests pass
- `api/handler/websocket.rs` — **subscribe BEFORE initial snapshot** (the
"everything needs ctrl-r" root cause: broadcasts in the snapshot→subscribe
gap were silently lost; a stale client never learned containers-scanned).
- `main.rs` — crash check now runs BEFORE writing the PID marker (**crash
recovery had never run on any node** — it always saw its own PID and
skipped); tracing default demoted debug→info (journal volume).
- `crash_recovery.rs` — PID-reuse guard (`process_is_archipelago`); new
**pending-boot-starts registry** (names queued for recovery/reconcile) with
writers in `recover_containers` + stack recovery.
- `server.rs` — scanner overlays Stopped/Exited → **Restarting** for
pending-boot-start ids (user ask: "status should be restarting if they are
being restarted"); `SCANNER_RESTARTING` ownership set so scanner-authored
Restarting resolves immediately instead of wedging in the 20-min
transitional-preserve.
- `container/prod_orchestrator.rs` — reconcile pass + `adopt_existing`
register/deregister pending boot-starts; LND pre-start hook passes detected
`bitcoin_host()` (Knots vs Core) into `lnd::ensure_config`; new
`fedimint-clientd` pre-start hook (mkdir + chown 1000:1000 of
`/var/lib/archipelago/fmcd` — self-heals the crash-loop).
- `container/lnd.rs` — `ensure_config(paths, rpc_pass, bitcoin_host)`;
bitcoind.rpchost no longer hardcoded `bitcoin-knots`; drift check rewrites
host changes; +unit test `ensure_config_repairs_bitcoin_host_drift`.
- `api/rpc/package/dependencies.rs` — bounded **dependency wait**
(`wait_for_install_deps`, 36×5s): installed-but-starting deps wait with
"Waiting for Bitcoin to start…" on the card; not-installed deps fail fast
with `DependencyGateError` marker; +5 unit tests.
- `api/rpc/package/install.rs`, `stacks.rs` — call sites wired to
`gate_install_deps` (lnd/electrumx/mempool/btcpay).
- `api/rpc/package/async_lifecycle.rs` — `DependencyGateError` removes the
optimistic entry (**no more phantom "Stopped" LND tile**) + pushes an Error
notification with the reason.
- `api/rpc/package/progress.rs` — `set_install_message` helper.
- `api/rpc/seed_rpc.rs` — `save_pending_seed_encrypted`; seed.restore also
stashes the mnemonic; `auth.rs` — **auth.setup persists the encrypted seed
backup** (recovery-phrase reveal previously failed on EVERY node because
nothing ever wrote `master_seed.enc`).
- `api/rpc/middleware.rs` — sanitizer allowlist extended (seed/2FA/auth
errors reach the user instead of "Check server logs"); +2 tests.
- `bitcoin_status.rs` — friendly status for "connection reset" (bitcoind
starting); raw URL/os-error chains no longer shown; +3 tests.
- `bootstrap.rs` — journald drop-in self-heal (OTA nodes get log caps);
bitcoin.conf printtoconsole heal. (Log-spam agent's work; verified.)
- `api/rpc/package/config.rs` — bitcoin args `-printtoconsole=0`.
### Manifests / scripts / configs
- `apps/lnd/manifest.yml` — BITCOIND_HOST now `derived_env {{BITCOIN_HOST}}`.
- `apps/bitcoin-knots/manifest.yml`, `apps/bitcoin-core/manifest.yml` —
`-printtoconsole=0` (90.6% of the journal was IBD UpdateTip spam;
debug.log in the datadir keeps full logs).
- `scripts/first-boot-containers.sh` — chown 1000:1000 of
`/var/lib/archipelago/fmcd` in BOTH fmcd blocks (root-owned dir was the
fedimint-clientd "Permission denied os error 13" crash-loop);
printtoconsole=0.
- `scripts/container-doctor.sh`, `scripts/reconcile-containers.sh` —
printtoconsole=0.
- `image-recipe/configs/journald-archipelago.conf` (NEW) — SystemMaxUse=500M,
rate limits; baked by ISO builder + bootstrap self-heal.
- `image-recipe/configs/nginx-archipelago.conf` — `/assets/` 404s no longer
cacheable (the `always` immutable header could pin a missing background for
a YEAR); HTTPS block gained the missing `/assets/` location (was silently
serving index.html as images).
- `image-recipe/configs/archipelago-kiosk.service` — MemoryMax 1500→2800M,
MemoryHigh 1200→2200M (kiosk was riding reclaim-throttle = the lag).
- `image-recipe/_archived/build-auto-installer-iso.sh` — kiosk launcher/service
now spliced from `image-recipe/configs/` at build time (was a stale inline
heredoc that force-disabled GPU); **+ `firmware-intel-graphics` +
`firmware-amd-graphics`** (Debian trixie split the i915 DMC blobs out of
firmware-misc-nonfree; the .81 kernel logged tgl_dmc missing).
### Frontend (neode-ui) — vue-tsc clean, vitest green
- `views/Login.vue` — Enter in field 1 → focus confirm; Enter in confirm →
submit; submit button always clickable (shows inline mismatch/length error
instead of being silently disabled); errors clear on input; **Restart
Onboarding needs a confirming second click** (5s window) — this button is
the likely cause of the "onboarding restarted after mismatch" report.
+`login.restartConfirm` key in en/es locales.
- `stores/sync.ts` — 30s staleness reconciliation (server.get-state) while
connected; already-connected fast path now refetches too.
- `composables/useContainersScanTimeout.ts` (NEW, +tests) — 20s escape hatch;
wired into `Apps.vue` / `Discover.vue` / `Marketplace.vue`; fresh empty node
reaches the real "no apps yet" empty state; "Checking…" can never persist.
- Backgrounds: 10 heaviest bg JPEGs → **WebP q90** (9.4MB→6.6MB; refs updated
in OnboardingWrapper/Dashboard/useRouteTransitions); 7 remaining images
stayed JPEG (WebP came out LARGER on those — noisy sources; deliberate).
- `public/assets/video/video-intro.mp4` — re-encoded CRF20 (SSIM 0.988) with
**+faststart** (moov was at EOF → browser had to download all 15MB before
playing = the intro lag). 12.7MB now, streams immediately.
- LND icon: stale dist artifact; any fresh `npm run build` ships
`app-icons/lnd.png` correctly.
## Verification done here
- `cargo build -p archipelago` + `cargo check` clean; targeted tests
(bitcoin_status, middleware sanitize, dep_wait, lnd, crash_recovery,
boot_reconciler, bitcoin_host, prod_orchestrator lnd hooks): **52 passed,
0 failed**. Full suite: **898 passed, 0 failed, 1 ignored** (22s).
- `npm run build` green; dist verified: 10 bg-*.webp present, `lnd.png`
icon present, `restartConfirm` string in bundle, optimized faststart
video (12,740,782 bytes) in place. Note: main had a latent build breaker
(unused template ref in `Web5ConnectedNodes.vue` from commit 8256fde1,
vue-tsc TS6133) — fixed here by removing the dead ref/binding; without
this fix `npm run build` fails on current main.
- vitest: new composable tests + related suites pass.
- `bash -n` clean on all touched scripts; nginx conf live-verified by agent
(200/404/cache headers on both HTTP+HTTPS blocks).
- ISO kiosk splice byte-verified against configs/ by agent simulation.
## NOT done / left for you
1. **Full test-suite run + gate**: run the complete `cargo test` and (after
deploy) `tests/lifecycle/run-gate.sh` ON .228 per CLAUDE.md before any tag.
2. **Frontend bundle grep before shipping** (per memory/feedback): verify new
strings (e.g. `restartConfirm`, `bg-home.webp`) in the built tarball.
3. **Diagnostics collector** (`data-dir-listing.txt` = 15MB of podman overlay
internals; dmidecode empty) — collector script wasn't found in this repo
(likely lives on-node or in the user's collection script); fix when found.
4. **podman healthcheck cgroup EPERM spam** (1,250 journal errors, healthchecks
unreliable fleet-wide) — real open bug, Quadlet-phase territory, NOT fixed.
5. **DP link-training failures on .81** (display corruption) — likely
cable/dock/port hardware; firmware fix may help; tell user to try another
cable/port if corruption recurs.
6. **LoRa/RNode onboarding surface** — never scoped; user may want it as a
feature (mesh device-found modal exists only on Mesh page post-login).
7. The concurrent audit agent's files (`docs/1.8.0-RELEASE-HARDENING-PLAN.md`,
`core/.../trust/*`, parts of `bootstrap.rs`) are ALSO uncommitted here —
coordinate before committing; don't mix attribution.
docs/archive/INSTALL-SCREENS-DESIGN.md
+2 -2
View File
@@ -92,9 +92,9 @@ Constraints: bash TUI only (no ncurses). ANSI colors available:
After reboot, open the Web UI from any device:
http://192.168.1.198
http://archipelago.local
SSH: ssh archipelago@192.168.1.198
SSH: ssh archipelago@archipelago.local
Password: archipelago
Web Login: password123
docs/archive/README.md
+2 -6
View File
@@ -7,20 +7,16 @@ docs whose feature has since shipped. They are kept for provenance and are
For current state, start at:
- `docs/UNIFIED-TASK-TRACKER.md` — what's open, priority-ordered
- `docs/PRODUCTION-MASTER-PLAN.md` — north star and workstream narrative
- `docs/architecture.md` — as-built system architecture
- `docs/ROADMAP.md` — public-facing roadmap
| File | What it was | Why archived |
|------|-------------|--------------|
| `SESSION-1.8.0-OTA-PROGRESS.md` | Session narrative of the 1.8.0 OTA work | Superseded by the unified task tracker |
| `HANDOVER-2026-07-02-iso-feedback.md` | One-shot handover for the ISO feedback bug-bash | All fixes merged |
| `rust-orchestrator-migration.md` | Design for migrating container lifecycle from bash to Rust | Migration complete — `prod_orchestrator.rs` + `boot_reconciler.rs` are the live system |
| `demo-deployment-design.md` | Design for the public demo sandbox | Demo shipped; `docs/demo-build-info.md` is the live ops doc |
| `app-registry-status-2026-06-21.md` | Per-app migration snapshot from node .228 @ v1.7.99-alpha | Point-in-time snapshot; headline findings (immich legacy, meshtastic present) no longer true |
| `security-code-audit-2026-03.md` | March 2026 security audit of v0.1.0 (33 findings) | Historical record; top findings since remediated (Argon2id, persisted sessions, image verification) |
| `architecture-review.html` | Generated interactive architecture guide (2026-03) | Stale generated artifact; describes an early crate/app layout |
| `lora-functionality.html` | Generated LoRa/mesh guide (2026-04) | Predates X3DH/double-ratchet, Reticulum transport, and mesh AI |
| `INSTALL-SCREENS-DESIGN.md` | Installer screen design solicitation | Installer implemented in `image-recipe/` |
| `three-mode-ui-design.md` | Design for the Pro/Easy/Chat three-mode UI | Fully implemented (`stores/uiMode.ts`, `EasyHome.vue`, `Chat.vue`, goals system) |
| `HANDOVER-2026-07-02-iso-feedback.md` | Session handover from the 2026-07-02 ISO feedback bug-bash | Completed session log |
| `SESSION-1.8.0-OTA-PROGRESS.md` | Working notes from the 1.8.0 OTA push | Completed session log |
docs/archive/SESSION-1.8.0-OTA-PROGRESS.md
-344
View File
@@ -1,344 +0,0 @@
# 1.8.0 OTA Session Progress
Updated: 2026-06-30
> **📋 Live day-to-day task tracker: `docs/UNIFIED-TASK-TRACKER.md`.** This doc is kept
> as the historical session-by-session log; open items were consolidated into the
> unified tracker on 2026-07-01 (several turned out already shipped — see that doc for
> current status instead of re-deriving it from the log below).
---
## ▶️▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (evening) — #17 deployed + verified on .198/.228
**#17 (3ccc / stock-peer E2E pill) is now built, deployed, and live-verified** on `.198` and
`.228` only (`.116` skipped per the hardware notice below — its radio is mid-reflash to RNode).
- Built release binary **sha `b1d695fc626a7382`** from the working tree (`cargo check` +
`cargo test -p archipelago mesh::` both green, 99 passed/0 failed/1 ignored, right before
building — tree was settled, no collision with the Reticulum agent's concurrent edits).
- Deployed via stop/swap/start to `.198` (192.168.1.198) and `.228` (192.168.1.228), sha256
confirmed matching on both, `systemctl is-active` = `active` on both (`.228` took its usual
~couple-minute convergence — heavy resilience node, unrelated bitcoind/fedimint container
startup noise in the logs during that window, no mesh errors).
- **Live-verified the actual fix**, not just deploy: on `.198`, `mesh.peers` shows
`"advert_name":"Meshtastic 3ccc", "pkc_capable":true`, and `mesh.send` to 3ccc
(`contact_id:1128152268`) now returns **`"encrypted":true`** — confirms the
`archy || peer_pkc_capable(contact_id)` TX fix is live, not just compiled.
- `.228`'s RPC password in memory (`password123`) was stale — user confirmed the correct
password is `ThisIsWeb54321@` (same as `.198`/`.116`, i.e. fully unified now). Re-verified via
RPC: `mesh.peers` shows 3ccc `pkc_capable:true`, and `mesh.send` to 3ccc returns
`"encrypted":true` — #17 confirmed live on `.228` too, not just `.198`.
**NOT yet done:** push commit to gitea-vps2 (still uncommitted in the working tree, by design —
shares the tree with the Reticulum agent's uncommitted work); user on-device confirmation that
the E2E pill actually renders in the Mesh UI for 3ccc.
---
## 🛠️ HARDWARE NOTICE 2026-06-30 (~16:30) — .116's Heltec V3 is being repurposed
**The Reticulum agent is reflashing .116's Heltec V3 (the board on `/dev/ttyUSB0`, currently
.116's live Meshtastic radio) to RNode firmware**, with explicit user approval, to unblock the
Reticulum Phase-0 hardware gates (real RNode needed; see `docs/RETICULUM-TRANSPORT-PROGRESS.md`).
This was user-confirmed specifically because it takes .116 offline as a Meshtastic radio.
**Effect on this workstream: do all on-device Meshtastic testing on .198 and .228 only — .116 no
longer has a Meshtastic-firmware radio attached once this lands.** `cargo check`/`cargo test
-p archipelago` were both confirmed clean (99/99 mesh tests) right before the reflash started, so
the earlier "wait for their edit to settle" blocker above is cleared — software-side it's safe to
build/test/deploy; only .116's *physical radio role* changed.
---
## ▶️▶️▶️ LIVE CHECKPOINT 2026-06-30 (later PM, ~15:50) — READ THIS FIRST IF RESUMING
**#17 (3ccc / stock-peer E2E pill) is CODE-COMPLETE in the working tree**, isolated
to `meshtastic.rs`/`protocol.rs`/`types.rs`/`mod.rs` as planned (no `session.rs`
transport-plumbing changes from this side):
- `ParsedContact.pkc_capable` (`protocol.rs`) + `MeshPeer.pkc_capable` (`types.rs`),
both `#[serde(default)]`/defaulted `false` at every construction site.
- `MeshtasticDevice::get_contacts()` now stamps `pkc_capable` per contact from the
existing `peer_is_pkc_capable(node_num)` seam (de-`allow(dead_code)`'d).
- `listener/session.rs::refresh_contacts` ORs the new value into `MeshPeer.pkc_capable`
(capability only grows, never cleared by a transient refresh) — this IS a touch of
session.rs, but additive/non-colliding with the Reticulum device-enum match arms
already there; did not touch transport plumbing/routing.
- `mod.rs::MeshService::send_message` now does `archy || self.peer_pkc_capable(contact_id)`
for the Sent-row `encrypted` flag (was `archy`-only before).
- Verified via `cargo check -p archipelago --bin archipelago` (clean, exit 0) **before**
the other agent's latest edit landed.
**NOT YET DONE:** rebuild release binary → redeploy 5 nodes → push → user on-device test
(same as #16, both still pending live verification).
**⚠️ BLOCKED right now — do not build/deploy/push until this clears:** the Reticulum
agent is actively mid-edit in the *same* working tree. A `cargo test` run right after
the clean `cargo check` above failed with a real (but transient, not mine) signature
mismatch: `session.rs::auto_detect_and_open` / `run_mesh_session` were observed with a
new `device_kind: Option<DeviceType>` param that `listener/mod.rs`'s call site didn't
have yet — a normal in-flight snapshot of their work, not a regression to fix here.
**Action on resume: re-run `cargo check` first; if it's clean, the other agent's edit
has settled and it's safe to proceed to build/test/deploy. If still broken, wait —
do not stash, revert, or patch their in-progress session.rs/listener/mod.rs changes**
(see memory `feedback_concurrent_agent_tree.md`). Also: building/deploying right now
would bundle their not-yet-finished `reticulum.rs` wiring into the binary — confirm
with the user before shipping a combined build, since only the meshtastic `#17` piece
has been asked for/owned by this session.
---
## ▶️▶️ LIVE CHECKPOINT 2026-06-30 (late PM) — READ THIS FIRST
**Fleet state:** all **5 test nodes** on binary **`38c456b0bacec3c4`** + frontend
**`Mesh-CAkPgvLo.js`**, `archipelago` active on each:
`.116`, `.198`, `.228` (LAN, archipelago@ + `~/.ssh/archipelago-deploy`),
`100.72.136.5`, `100.89.209.89` (Tailscale, same key — installed this session;
SSH user `archipelago` / pw `ThisIsWeb54321@`; NOPASSWD sudo on all 5).
**Shipped this session (commit `12e7990b` on `main`, pushed to gitea-vps2):**
- ✅ **#16 public-channel routing** — inbound Meshtastic text to `BROADCAST_NUM`
now files under the **public channel thread** (contact_id `u32::MAX - idx`),
attributed to its real sender, instead of polluting per-sender DM threads.
Directed text (`to == our node`) still routes to the DM thread (regression test
`packet_to_inbound_frame_directed_dm_stays_a_contact_message`). `send_channel_text`
now sets `MeshPacket.channel` so archy TX's on channel 0 (public).
Code: `meshtastic.rs` (`packet_to_inbound_frame`, `parse_mesh_packet` to/channel,
`send_channel_text`), `protocol.rs` (`RESP_MESHTASTIC_CHANNEL_TEXT = 0x70`),
`listener/frames.rs` (handler + sender attribution), `Mesh.vue` (`senderLabelFor`).
Tests green (95 mesh tests). **Pending: user on-device test with the radios.**
**Push access:** `main` is a PROTECTED branch on gitea-vps2. Direct push uses the
dedicated **`ai`** account via remote **`gitea-ai`** (`git push gitea-ai main`).
See memory `reference_gitea_ai_push_account.md`.
**Coordination:** another agent owns **Reticulum** (`reticulum-daemon/` + Rust
transport wiring). DO NOT touch `mesh/listener/session.rs` transport plumbing or
`mod.rs` routing in ways that collide. Keep #17 work isolated to `meshtastic.rs`
RX/TX + (if needed) the sent-row encrypted flag.
### ✅ CODE-COMPLETE (not yet deployed/tested live) — #17 (3ccc / stock-peer E2E pill)
Goal: DMs **to and from** a PKC-capable stock peer (3ccc, NodeInfo public_key
key_len=32 confirmed) must show the E2E pill.
- **RX side is already correct:** `parse_mesh_packet` reads `public_key` (field 16)
+ `pki_encrypted` (field 17) per the MeshPacket proto; the directed-DM RX path
promotes to `RESP_CONTACT_MSG_V3_E2E` when `pki_encrypted`. (Verify live.)
- **TX bug (root cause) — FIXED:** `mod.rs::send_message` now records the Sent row
with `encrypted = archy || peer_pkc_capable(contact_id)`. `peer_is_pkc_capable`
(meshtastic.rs) is wired out via `get_contacts()` → `ParsedContact.pkc_capable` →
`refresh_contacts` (session.rs) → `MeshPeer.pkc_capable` → `MeshService::peer_pkc_capable`.
See the LIVE CHECKPOINT at the top of this file for the exact touch points.
- NEXT STEP when resuming: confirm `cargo check` is clean (the other agent's
Reticulum work shares this tree and may be mid-edit — see top checkpoint), then
rebuild → redeploy 5 nodes → push → user test (same pending step as #16).
**Remaining open after #17:** #12 (provisioning robustness — HOLD, session.rs churn
risks reticulum collision), #8 (Device-tab settings panel + reboot button — RPC
`mesh.reboot-radio` already exists), #6 (onboarding modal), #7 (.116 re-verify),
#14 (RSSI/SNR per-contact indicator), #15 (peer-location map, POSITION_APP portnum=3).
---
## ▶️ RESUME HERE — archy↔archy LoRa (2026-06-30 PM) — READ FIRST
**Goal:** archy↔archy text over Meshtastic LoRa must DELIVER and show the E2E pill,
identical in off-grid and normal mode. Test bed = `.116` / `.198` / `.228` (all EU_868).
Don't touch the federation/FIPS path.
### ✅✅✅ SOLVED 2026-06-30 — archy↔archy LoRa WORKS (delivery + E2E pill + identity)
VERIFIED: `.198→.228` directed DM → `.228` row `RECEIVED enc=True peer="Arch Optiplex"`.
All three nodes (.116/.198/.228) now hear each other + stock peer 3ccc. Deployed binary
**`737b16c3235b`** active on all three. Fix source **COMMITTED as `a57ae388`** on `main`
(not yet pushed to gitea-vps2/origin).
**THE fix (receive stream):** archy ignored `FromRadio.rebooted` (field 8). Every config
write reboots the radio → firmware PhoneAPI resets to `STATE_SEND_NOTHING` and stops
streaming received packets until the client re-sends `want_config`. archy never did →
went deaf to inbound (that's why old messages only arrived after a full restart = fresh
want_config). Fix: handle `FROM_RADIO_REBOOTED` → set `pending_reinit` → re-send
want_config; plus a 10s keepalive heartbeat (insurance vs 15-min idle serial close) and
a pinned `modem_preset=LONG_FAST` so all radios share frequency. Combined with the earlier
E2E send fix (plain TEXT_MESSAGE_APP DM, firmware PKC) this closes archy↔archy LoRa.
**Open follow-ups:** #A surface received msgs under archy identity in all UI views; #6
device-onboarding modal; #8 Device-tab settings panel; #7 re-verify .116 in rotation;
#12 make modem_preset authoritative + hot-swap re-binding + RX-stall watchdog;
#14 signal-strength (RSSI/SNR) indicator per contact (from MeshPacket rx_rssi/rx_snr);
#15 map view plotting peer locations where shared (Meshtastic POSITION_APP portnum=3
lat/lon). See the resume memory `project_session_resume_2026_06_30_lora.md` for the full
task list.
### (historical) earlier TL;DR — RF-layer suspicion, now RESOLVED by the reboot-recovery fix
The **archy software is correct and deployed.** The blocker was at the
**radio/RF layer: the three radios are not hearing each other over the air at all.** No
amount of archy code change will fix that until the radios actually RF-link. **Resume by
testing the radios directly at home (Meshtastic phone app over Bluetooth) — see "DO THIS
FIRST AT HOME" below.** ← this turned out to be the want_config resubscribe bug above.
### What is DONE and deployed (commit pending — see below)
- **E2E send fix** (`core/archipelago/src/mesh/mod.rs` `send_message`, ~L1542): archy↔archy
plain chat text is now sent as a **native `TEXT_MESSAGE_APP` DM** (firmware PKC-encrypts
it E2E), NOT wrapped in our binary typed envelope. Archy peers' Sent rows are marked
`encrypted=true` so the pill shows. Rich typed msgs still use `send_typed_wire`. This was
the original root-cause fix (envelope-wrapped text silently broke archy↔archy LoRa).
- **NEW: software radio-reboot** end-to-end, so a wedged/RX-deaf radio can be rebooted
without physical access (and for the Device-tab settings panel the user requested):
- `meshtastic.rs`: `reboot(seconds)` driver method + `ADMIN_REBOOT_SECONDS_FIELD = 97`
(verified vs meshtastic/protobufs admin.proto — `set_owner=32/set_channel=33/set_config=34`
matched our existing constants, confirming the proto read).
- `listener/mod.rs`: `MeshCommand::RebootRadio { seconds }`.
- `listener/session.rs`: device-enum `reboot()` dispatch (Meshtastic only) + handler arm.
- `mesh/mod.rs`: `MeshService::reboot_radio(seconds)`.
- `api/rpc/mesh/messaging.rs`: `handle_mesh_reboot_radio` → RPC **`mesh.reboot-radio`**
`{seconds?}` (default 2); dispatcher arm in `api/rpc/dispatcher.rs`.
- `cargo check` passes. Built release **sha `ba4aed590027690d`** and DEPLOYED + active on
`.116/.198/.228`. The RPC works (`{"reboot":true,"seconds":2}`).
- ⚠️ **Caveat:** when called, archy logged "Sent Meshtastic radio reboot" but the radio did
**not** visibly reboot afterward (no config re-stream). Either field 97 is still off, or
newer firmware requires an admin session passkey even over local serial, or the USB serial
stayed open through the 2s reboot so no reconnect was logged. **Needs on-device verification.**
### The hard evidence (why "nothing works")
- Directed DM tests `.198→.228` AND `.116→.228` (neither path reflashed): sender logs
`Sent plain native DM dest=30d258436d65 part=1 total=1` and RPC returns `sent:true,
encrypted:true`, but `.228` logs **nothing** — packet never reaches archy from the radio.
- A raw broadcast from `.198` (`mesh.broadcast`) was accepted by its radio but **not heard**
by `.228`/`.116`.
- In an 8-minute window, **all three nodes received 0 inbound OTA packets from any other node.**
Each only logs its OWN once-a-minute `Broadcast Meshtastic NodeInfo advert` + local TX
`field=11` queue-status. `.228 mesh.status` = `messages_received:1` total.
- `.198`'s radio is alive and transmitting NodeInfo every 60s — so it's not dead; it's that
**reception is broken on the receivers.** A radio cannot drop a broadcast AND a unicast to
its own node number while config matches, unless it simply isn't on the same airwaves.
- archy provisioning is correct & identical across nodes (read back from device): PRIMARY =
public LongFast (`name="" psk_len=1`), SECONDARY = `archipelago`, region=3 (EU_868). Admin
field constants verified. The send path hands the radio a correct unicast MeshPacket
(`to`=node, want_ack, hop_limit=3, plaintext `decoded` for the firmware to PKC-encrypt).
### PRIME SUSPECT (software-fixable) — modem-preset / frequency mismatch
archy only ever writes `region` + `use_preset` and **never explicitly pins `modem_preset`**
(it parses region but not preset; `set_lora_region` relies on the LongFast default). If ANY
radio has a non-default modem preset / frequency slot persisted (e.g. set via the Meshtastic
app, or a different factory default after the `.198` reflash), the radios are on **different
airwaves despite identical channel name + region**, and archy would never correct it.
### DO THIS FIRST AT HOME (decisive, ~2 min, only the user can do it)
Open the **Meshtastic phone app over Bluetooth** (works alongside archy's USB serial) on each
of `.116/.198/.228` and check:
1. Do the 3 nodes **see each other** in the node list (recent "heard")? → if NO, they're not
RF-reaching (preset/freq/antenna/range).
2. Do all 3 show the **same** Modem preset (LongFast), Region (EU_868), Frequency slot, and
the same PRIMARY channel? → any difference = the cause.
This single test separates "archy misconfigures the radios" from "radios physically can't
reach each other."
### THEN — the archy fix to apply (if preset/config differs)
Make archy **authoritatively write the full LoRaConfig** and force re-provision so all radios
converge: in `core/archipelago/src/mesh/meshtastic.rs::set_lora_region` (and its
caller/guard `ensure_lora_region` ~L304), explicitly set `modem_preset = LONG_FAST (0)` as a
field in the LoRaConfig (it's currently omitted/defaulted), and make the startup provision
path rewrite LoRa config when the preset doesn't match, then reboot the radio (use the new
`mesh.reboot-radio`). Also verify the `mesh.reboot-radio` actually reboots the radio
on-device (the caveat above).
### TEST RECIPE (works on each node)
- RPC helper used this session: a node-side `rpc.sh` that logs in (password
`ThisIsWeb54321@`), grabs the `csrf_token` cookie, echoes it as `X-CSRF-Token`, and POSTs to
`http://127.0.0.1:5678/rpc/v1`. Recreate it or run archy's RPC directly. Methods:
`mesh.peers`, `mesh.status`, `mesh.messages`, `mesh.send {contact_id,message}`,
`mesh.broadcast`, `mesh.reboot-radio {seconds}`.
- **LoRa contact ids:** `.116=1135977788` (prefix `3ca5b543`), `.198=3677050140` (`db2b551c`),
`.228=1129894448` (prefix `30d25843`), stock `3ccc=1128152268`.
- **Link health check (run on each node):** look for inbound `from=Some("!...")` lines in
`journalctl -u archipelago` that are NOT the node's own `Broadcast ... NodeInfo advert`. If
zero across all nodes → RF link is down (the current state).
- **E2E success criteria:** send `.198→.228`, the marker appears in `.228` `mesh.messages` as
an inbound row with `encrypted:true` / `transport:"lora"`, AND `.116↔.228` likewise.
### DEPLOY / BUILD RECIPE
- Build: from `core/`, `CARGO_TARGET_DIR=/tmp/archy-hotfix-target CARGO_INCREMENTAL=0 cargo
build --release -p archipelago --bin archipelago`. (If `rust-lld: undefined hidden symbol`,
it's incremental cache — `CARGO_INCREMENTAL=0` fixes it.)
- SSH key `~/.ssh/archipelago-deploy` is authorized on `.116/.198/.228`. SSH/UI/RPC password
`ThisIsWeb54321@`. Per node: scp the binary, `sudo systemctl stop archipelago` →
`kill -9 $(pgrep -x archipelago)` → `install -m0755` to `/usr/local/bin/archipelago` →
`systemctl start archipelago`. Verify by `sha256sum` match + `systemctl is-active`.
- **Current deployed sha on all 3 = `ba4aed590027690d`** (the reboot-enabled build).
### Fleet state (as of 2026-06-30 PM)
- All 3 nodes on binary `ba4aed59`, active. Off-grid mode currently OFF (`mesh_only:false`).
- `.198` radio was reflashed to factory `firmware-heltec-v3-2.7.26` (recovered from corrupt
NVS); region EU_868 persists. Its archy identity is NOT re-bound on `.228` (`.228` shows
`.198` as raw radio "Meshtastic 551c", `arch_pubkey_hex` absent) because `.228` hasn't heard
`.198`'s identity broadcast — a downstream symptom of the dead RF link, not a separate bug.
- The radios are powered & each transmitting; they are simply not hearing each other.
### Deferred UI (after LoRa works)
- Device-tab **settings panel** (gear/desktop) — host the "Reboot radio" button there; calls
`mesh.reboot-radio`. Scoping done: add to the Mesh.vue actions row (mirrors Broadcast/Off-Grid
buttons) + a `rebootRadio()` method in `neode-ui/src/stores/mesh.ts`. See `Mesh.vue` ~L1484
actions row and `mesh.ts` ~L373 `broadcastIdentity()` pattern.
- Device-onboarding modal (detect plugged-in radio).
---
Current scope:
- Preserve existing mesh work: E2E indicators, FIPS/Tor transport indicators, typed-message paths, Meshtastic region/channel provisioning, and dirty Meshtastic receive-attempt changes.
- Take over the `3ccc` stock Meshtastic peer bug: LoRa text from `3ccc` to Archipelago `.116` does not surface in `mesh.messages`.
- Keep release-gate fixes already made in this session.
Local gate status so far:
- `cargo test -p archipelago --bin archipelago`: green, 849/849 after Meshtastic fixes.
- `python3 scripts/check-app-catalog-drift.py --release --strict`: green.
- `npm run type-check`: green.
Key changes made so far:
- Added cascade uninstall progress truthfulness assertion to `tests/lifecycle/bats/cascade-uninstall.bats`.
- Fixed release catalog drift filters and regenerated catalog metadata.
- Fixed invalid `apps/fedimint-clientd/manifest.yml` `cpu_limit` schema value.
- Updated stale/tight Rust tests without changing production behavior.
Remaining non-automatable / operational gates:
- Workstream B signing is blocked on the offline `RELEASE_MASTER_MNEMONIC`; code + runbook exist, but the publisher must pin/sign the release-root catalog.
- Phase-3 Quadlet backend rollout is implemented behind `use_quadlet_backends` and default-off. The gate skip-passes until explicitly enabled on a node; flipping it fleet-wide requires a coordinated flag rollout plus backend reinstall/migration verification.
- `.116` read-only `use-quadlet-backends-install.bats`: 6/6 skip-clean; no backend `.container` units, so Phase-3 is not active on that node.
- Release metadata still says `1.7.99-alpha` in `releases/manifest.json`; changelog top is `v1.8.00-alpha`. Cutting an actual 1.8.0 OTA requires an explicit version/manifest update.
Do not discard:
- `core/archipelago/src/mesh/listener/decode.rs`
- `core/archipelago/src/mesh/listener/session.rs`
- `core/archipelago/src/mesh/meshtastic.rs`
3ccc bug current hypothesis:
- The prior attempted Meshtastic fix added a hard stale-packet filter using `rx_time`.
- Stock Meshtastic radios without GPS/RTC can report tiny nonzero epoch values until time sync.
- That would make live `3ccc` packets look older than 10 minutes and get dropped before `mesh.messages`.
- Current patch treats implausibly early `rx_time` values as unknown rather than stale.
.116 live validation after 2026-06-30 hotfix:
- `.116` reachable by SSH; `archipelago` active; `/dev/mesh-radio -> ttyUSB0` attached.
- Current canary deploy is commit `b4531bb4`; backend sha
`4ab53e539d89679ef664401a9a57996267772fed02327abc2912c3e77543acbf`; frontend bundle
`index-YOAeJF7w.js` / `Mesh-BSAo88jN.js`.
- `main` pushed to `gitea-vps2`.
- RPC on `.116`:
- `transport.status` currently reports `mesh_only:false` (off-grid mode is not enabled unless
the user toggles it).
- `mesh.status` reports Meshtastic connected: `device_type:"meshtastic"`,
`self_node_id:1135977788`, `peer_count:13`.
- Recent `.116` -> `3ccc` sent rows are stored with real 2026 timestamps and `transport:"lora"`.
- UI/backend fixes included in `b4531bb4`:
- `transportLabel("lora")` displays **LoRa**.
- mesh sends refetch messages after send so transport pills settle without browser refresh.
- off-grid mode blocks the mesh-chat FIPS/Tor federation fallback and forces LoRa-only sends;
banner text is `Tor/FIPS disabled - LoRa only`.
- empty mesh-chat placeholder opacity reduced.
- Meshtastic diagnostics now identify the remaining blocker:
- 3ccc NodeInfo is discovered:
`Meshtastic peer is PKC-capable (NodeInfo public_key) node=1128152268 key_len=32`.
- Bytes from stock Meshtastic text reach `.116`, but the custom parser rejects the packet:
`Meshtastic FromRadio.packet did not parse into a decoded MeshPacket len=73 head=0dcc3c3e43153ca5b5432a16df56cbed`.
- Non-text packets decode and are ignored with port numbers (`portnum=3/4/5`), so the serial
read path is alive. Resume inside `core/archipelago/src/mesh/meshtastic.rs::parse_mesh_packet`.
- LoRa is therefore **not fully fixed** yet: stock `3ccc` -> `.116` text does not surface in
`mesh.messages`, and `.116` -> `3ccc` still needs user-visible confirmation in the Meshtastic app.
docs/archive/architecture-review.html
-2523
View File
File diff suppressed because it is too large Load Diff
docs/archive/demo-deployment-design.md
+3 -4
View File
@@ -7,8 +7,7 @@ secrets/backend. Deployed via **Portainer**, mock-data driven, with working file
storage and a testnet-flavored Bitcoin sandbox so visitors can play freely.
See also: `neode-ui/mock-backend.js` (existing mock), `docker-compose.demo.yml`
(existing demo stack), `MEMORY → reference_neode_ui_dev_testing`,
`MEMORY → reference_ovh_168_mirror` (Portainer/registry host).
(existing demo stack).
---
@@ -46,7 +45,7 @@ layer**.
CI: build archy-demo-web + archy-demo-backend
│ push :demo / :latest
▼
registry (146.59.87.168:3000 / vps2)
registry (source.archipelago-foundation.org / vps2)
│ Portainer webhook / re-pull
▼
archy-demo (public repo — tiny)
@@ -162,7 +161,7 @@ Today filebrowser upload/delete/rename are 200-OK no-ops.
1. **Demo host** — which Portainer instance (OVH `.168`? a dedicated VPS)? Public
DNS + TLS for `demo.<domain>`?
2. **Registry for `:demo` images** — `146.59.87.168:3000` vs vps2; public-pull or
2. **Registry for `:demo` images** — `source.archipelago-foundation.org` vs vps2; public-pull or
creds baked into Portainer?
3. **Session TTL + concurrency cap** — concrete numbers (30 min / N sessions / 50 MB)?
4. **Chat in the demo** — enable Claude chat (needs key + budget cap) or stub it?
docs/archive/lora-functionality.html
-899
View File
@@ -1,899 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Archipelago — LoRa & Mesh Functionality Guide</title>
<style>
:root {
--bg: #000000;
--glass-card: rgba(0, 0, 0, 0.65);
--glass-dark: rgba(0, 0, 0, 0.35);
--glass-darker: rgba(0, 0, 0, 0.6);
--glass-border: rgba(255, 255, 255, 0.18);
--glass-highlight: rgba(255, 255, 255, 0.22);
--glass-blur: 18px;
--glass-blur-strong: 24px;
--shadow-glass: 0 8px 24px rgba(0, 0, 0, 0.45);
--shadow-glass-inset: inset 0 1px 0 rgba(255, 255, 255, 0.22);
--text: rgba(255, 255, 255, 0.9);
--text-muted: rgba(255, 255, 255, 0.6);
--accent: #fb923c;
--accent-dim: rgba(251, 146, 60, 0.15);
--green: #4ade80;
--green-dim: rgba(74, 222, 128, 0.15);
--red: #ef4444;
--red-dim: rgba(239, 68, 68, 0.12);
--blue: #3b82f6;
--blue-dim: rgba(59, 130, 246, 0.12);
--yellow: #facc15;
--yellow-dim: rgba(250, 204, 21, 0.12);
--purple: #a78bfa;
--purple-dim: rgba(167, 139, 250, 0.12);
--radius: 16px;
--radius-sm: 12px;
--transition: 0.3s cubic-bezier(0.4, 0, 0.2, 1);
}
* { margin: 0; padding: 0; box-sizing: border-box; }
html { scroll-behavior: smooth; }
body {
font-family: 'Avenir Next', system-ui, -apple-system, sans-serif;
background: var(--bg);
color: var(--text);
line-height: 1.7;
font-size: 16px;
}
nav {
position: fixed;
top: 0; left: 0;
width: 280px;
height: 100vh;
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur-strong));
-webkit-backdrop-filter: blur(var(--glass-blur-strong));
border-right: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass);
overflow-y: auto;
padding: 24px 0;
z-index: 100;
scrollbar-width: thin;
scrollbar-color: rgba(255,255,255,0.15) transparent;
}
nav .logo { padding: 0 24px 20px; margin-bottom: 16px; }
nav .logo h1 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 18px; font-weight: 700;
color: var(--accent); letter-spacing: -0.02em;
}
nav .logo p { font-size: 12px; color: var(--text-muted); margin-top: 4px; }
nav .nav-section {
padding: 12px 16px 4px;
font-size: 10px; font-weight: 700;
text-transform: uppercase;
letter-spacing: 0.1em;
color: var(--text-muted);
}
nav a {
display: block;
padding: 6px 24px;
color: var(--text-muted);
text-decoration: none;
font-size: 13px;
transition: all var(--transition);
border-left: 2px solid transparent;
}
nav a:hover, nav a.active {
color: var(--text);
background: rgba(255, 255, 255, 0.06);
border-left-color: var(--accent);
}
main {
margin-left: 280px;
max-width: 960px;
padding: 48px 48px 120px;
}
h2 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 28px; font-weight: 700;
margin: 64px 0 8px;
padding-top: 24px;
color: var(--text);
letter-spacing: -0.02em;
}
h2:first-of-type { margin-top: 0; }
h3 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 20px; font-weight: 600;
margin: 40px 0 12px;
color: var(--text);
}
h4 {
font-size: 16px; font-weight: 600;
margin: 24px 0 8px;
color: var(--accent);
}
p { margin: 8px 0 16px; color: var(--text); }
ul, ol { margin: 8px 0 16px 24px; color: var(--text); }
li { margin: 4px 0; }
.subtitle {
font-size: 15px;
color: var(--text-muted);
margin-bottom: 32px;
}
.hero { text-align: center; padding: 48px 0 56px; margin-bottom: 24px; }
.hero h1 {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 42px; font-weight: 800;
background: linear-gradient(135deg, var(--accent), #f59e0b);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
letter-spacing: -0.03em;
}
.hero .tagline {
font-size: 18px;
color: var(--text-muted);
margin: 12px auto 0;
max-width: 640px;
}
.hero .meta {
margin-top: 20px;
display: flex; gap: 16px;
justify-content: center; flex-wrap: wrap;
}
.hero .meta span {
font-size: 12px;
padding: 4px 12px;
border-radius: 999px;
background: var(--glass-dark);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
color: var(--text-muted);
}
.card {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 24px;
margin: 16px 0;
}
.card-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
gap: 16px;
margin: 16px 0;
}
.card-sm {
background: var(--glass-darker);
backdrop-filter: blur(var(--glass-blur-strong));
-webkit-backdrop-filter: blur(var(--glass-blur-strong));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 16px 20px;
transition: transform var(--transition), box-shadow var(--transition);
}
.card-sm:hover {
transform: translateY(-2px);
box-shadow: 0 12px 32px rgba(0, 0, 0, 0.6), inset 0 1px 0 rgba(255, 255, 255, 0.25);
}
.card-sm h4 { margin: 0 0 6px; font-size: 14px; }
.card-sm p { font-size: 13px; color: var(--text-muted); margin: 0; }
.badge {
display: inline-block;
font-size: 11px; font-weight: 600;
padding: 2px 10px;
border-radius: 999px;
text-transform: uppercase;
letter-spacing: 0.05em;
}
.badge-green { background: var(--green-dim); color: var(--green); }
.badge-red { background: var(--red-dim); color: var(--red); }
.badge-yellow { background: var(--yellow-dim); color: var(--yellow); }
.badge-blue { background: var(--blue-dim); color: var(--blue); }
.badge-purple { background: var(--purple-dim); color: var(--purple); }
.badge-accent { background: var(--accent-dim); color: var(--accent); }
table {
width: 100%;
border-collapse: separate;
border-spacing: 0;
margin: 16px 0;
font-size: 13px;
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
border-radius: var(--radius-sm);
overflow: hidden;
box-shadow: var(--shadow-glass);
}
th {
text-align: left;
padding: 10px 14px;
background: rgba(0, 0, 0, 0.4);
color: var(--text-muted);
font-weight: 600;
font-size: 11px;
text-transform: uppercase;
letter-spacing: 0.05em;
border-bottom: 1px solid var(--glass-border);
}
td {
padding: 10px 14px;
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
vertical-align: top;
}
tr:last-child td { border-bottom: none; }
tr:hover td { background: rgba(255, 255, 255, 0.04); }
code {
font-family: 'Menlo', 'Monaco', 'Courier New', monospace;
font-size: 13px;
background: rgba(0, 0, 0, 0.4);
padding: 2px 6px;
border-radius: 4px;
color: var(--accent);
}
pre {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
border-radius: var(--radius-sm);
padding: 20px;
overflow-x: auto;
margin: 16px 0;
font-size: 13px;
line-height: 1.6;
box-shadow: var(--shadow-glass);
}
pre code { background: none; padding: 0; color: var(--text); }
.diagram {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur-strong));
-webkit-backdrop-filter: blur(var(--glass-blur-strong));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 24px;
margin: 20px 0;
overflow-x: auto;
font-family: 'Menlo', 'Monaco', monospace;
font-size: 13px;
line-height: 1.5;
color: var(--text-muted);
white-space: pre;
}
.diagram .highlight { color: var(--accent); font-weight: 600; }
.diagram .green { color: var(--green); }
.diagram .blue { color: var(--blue); }
.diagram .red { color: var(--red); }
.diagram .purple { color: var(--purple); }
.callout {
background: var(--glass-card);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
border-radius: var(--radius-sm);
padding: 16px 20px;
margin: 16px 0;
font-size: 14px;
border-left: 3px solid;
box-shadow: var(--shadow-glass);
}
.callout-info { border-color: var(--blue); }
.callout-warn { border-color: var(--yellow); }
.callout-danger { border-color: var(--red); }
.callout-success { border-color: var(--green); }
.callout-learn {
border-color: var(--purple);
background: rgba(167, 139, 250, 0.06);
position: relative;
padding-top: 32px;
}
.callout-learn::before {
content: 'Layman Analogy';
position: absolute;
top: 10px; left: 20px;
font-size: 10px;
font-weight: 700;
letter-spacing: 0.1em;
text-transform: uppercase;
color: var(--purple);
}
.callout strong { display: block; margin-bottom: 4px; }
.score-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));
gap: 12px;
margin: 20px 0;
}
.score-card {
background: var(--glass-darker);
backdrop-filter: blur(var(--glass-blur));
-webkit-backdrop-filter: blur(var(--glass-blur));
border: 1px solid var(--glass-border);
box-shadow: var(--shadow-glass), var(--shadow-glass-inset);
border-radius: var(--radius);
padding: 16px;
text-align: center;
transition: transform var(--transition);
}
.score-card:hover { transform: translateY(-2px); }
.score-card .score {
font-family: 'Montserrat', 'Avenir Next', sans-serif;
font-size: 28px; font-weight: 800;
margin: 4px 0;
color: var(--accent);
}
.score-card .label {
font-size: 11px;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.05em;
}
hr {
border: none;
border-top: 1px solid var(--glass-border);
margin: 48px 0;
}
@media (max-width: 900px) {
nav { display: none; }
main { margin-left: 0; padding: 20px; }
.hero h1 { font-size: 32px; }
}
</style>
</head>
<body>
<nav>
<div class="logo">
<h1>Archipelago</h1>
<p>LoRa &amp; Mesh Guide</p>
</div>
<div class="nav-section">Overview</div>
<a href="#intro">Introduction</a>
<a href="#layman">What is LoRa?</a>
<a href="#why">Why Archipelago uses it</a>
<div class="nav-section">Stack</div>
<a href="#hardware">Hardware &amp; Firmware</a>
<a href="#serial">USB Serial Transport</a>
<a href="#wire">Wire Format</a>
<a href="#crypto">Encryption Layers</a>
<a href="#fragmentation">Fragmentation</a>
<div class="nav-section">Routing</div>
<a href="#dual-transport">Dual Transport</a>
<a href="#addressing">Addressing</a>
<a href="#synthetic">Federation Contacts</a>
<div class="nav-section">Messages</div>
<a href="#msg-overview">All 23 Types</a>
<a href="#msg-text">Text / Reply / Edit</a>
<a href="#msg-social">Reactions &amp; Receipts</a>
<a href="#msg-content">Content / Files</a>
<a href="#msg-bitcoin">Bitcoin &amp; Lightning</a>
<a href="#msg-safety">Alerts &amp; Presence</a>
<a href="#msg-identity">Identity &amp; Keys</a>
<div class="nav-section">Operations</div>
<a href="#rpc">RPC API</a>
<a href="#ui">User Interface</a>
<a href="#listener">Listener Loop</a>
<a href="#files">File Map</a>
</nav>
<main>
<section class="hero">
<h1>LoRa &amp; Mesh Functionality</h1>
<p class="tagline">How Archipelago sends encrypted messages, Bitcoin transactions, and emergency alerts over long-range radio when the internet is gone.</p>
<div class="meta">
<span>Meshcore Companion USB</span>
<span>Double Ratchet E2E</span>
<span>23 Message Types</span>
<span>160-byte LoRa Frame</span>
</div>
</section>
<h2 id="intro">Introduction</h2>
<p>This document explains Archipelago's mesh subsystem — the code under <code>core/archipelago/src/mesh/</code> that lets nodes talk to each other over <strong>LoRa radio</strong> instead of (or alongside) the internet. It covers every message type, the transport layer that carries it, the cryptography that protects it, and the code paths that glue it all together.</p>
<p>The goal: give you a mental model that works both ways. If you're an engineer, you can read this and know exactly which bytes get put on the wire for a given RPC call. If you're not, the purple "Layman Analogy" boxes translate each piece into familiar metaphors.</p>
<h2 id="layman">What is LoRa? <span class="badge badge-purple">Layman</span></h2>
<div class="callout callout-learn">
<strong>Think of LoRa as a whisper that travels 10 kilometers.</strong>
Normal Wi-Fi is a shout: loud, fast, lots of data, but only a few rooms away. LoRa is the opposite — a tiny, slow whisper that can cross an entire city because it's so narrow and patient that it slips through walls, trees, and hills. The tradeoff: you can only whisper about <strong>160 bytes</strong> at a time, and each whisper takes a second or two to complete.
</div>
<p>Technically, LoRa (Long Range) is a proprietary radio modulation by Semtech that uses <em>chirp spread spectrum</em> (CSS). It operates in unlicensed ISM bands (915 MHz in the Americas, 868 MHz in Europe) and trades bandwidth for sensitivity, allowing receivers to decode signals below the noise floor. Typical line-of-sight range is 5–15 km with a simple antenna; data rates are 0.3–50 kbps.</p>
<p>Archipelago does not talk to a LoRa chipset directly. Instead it delegates to a small USB-attached device running <strong>Meshcore firmware</strong>, which handles the radio, the mesh routing, and the store-and-forward queue. Archipelago speaks to that device over USB serial.</p>
<h2 id="why">Why Archipelago uses it</h2>
<div class="card-grid">
<div class="card-sm">
<h4>Off-grid safety</h4>
<p>Dead-man switch and emergency alerts reach family without cell coverage.</p>
</div>
<div class="card-sm">
<h4>Censorship resistance</h4>
<p>No ISP, no DNS, no TLS termination — just radio waves between nodes.</p>
</div>
<div class="card-sm">
<h4>Bitcoin when internet is down</h4>
<p>Relay signed transactions and Lightning payments through on-grid peers.</p>
</div>
<div class="card-sm">
<h4>Truly peer-to-peer chat</h4>
<p>Text, replies, reactions, read-receipts — Telegram-quality UX, zero servers.</p>
</div>
</div>
<hr>
<h2 id="hardware">Hardware &amp; Firmware</h2>
<p>Archipelago expects a Meshcore-compatible radio board plugged into USB. The firmware handles RF, mesh forwarding, and contact management; Archipelago handles encryption, message types, and UI.</p>
<table>
<thead><tr><th>Component</th><th>Role</th><th>Examples</th></tr></thead>
<tbody>
<tr><td><strong>MCU</strong></td><td>Runs Meshcore firmware, talks USB serial</td><td>ESP32, nRF52840</td></tr>
<tr><td><strong>Radio</strong></td><td>Semtech LoRa transceiver</td><td>SX1262, SX1276</td></tr>
<tr><td><strong>Board</strong></td><td>MCU + radio + USB + antenna</td><td>Heltec V3, T-Beam, RAK WisBlock, Station G2</td></tr>
<tr><td><strong>Firmware</strong></td><td>Mesh routing + Companion USB protocol</td><td>Meshcore</td></tr>
<tr><td><strong>Connection</strong></td><td>USB CDC-ACM serial</td><td><code>/dev/mesh-radio</code> (udev symlink), <code>/dev/ttyUSB*</code>, <code>/dev/ttyACM*</code></td></tr>
<tr><td><strong>Link params</strong></td><td>115200 baud, 8N1</td><td>Set in <code>mesh/serial.rs</code></td></tr>
</tbody>
</table>
<div class="callout callout-learn">
<strong>It's a modem.</strong> Exactly like a 56k modem from the '90s plugged into your serial port, except the other end of the wire is a radio mesh network instead of a phone line. Archipelago tells it "send this to contact X", and it figures out which radios to hop through.
</div>
<h2 id="serial">USB Serial Transport</h2>
<p>Every byte in and out of the radio is wrapped in a framed serial protocol. The host speaks with <code>'&lt;'</code> and listens for <code>'&gt;'</code>.</p>
<div class="diagram">Host → Device: <span class="highlight">0x3C</span> '&lt;' │ <span class="blue">len_lo len_hi</span> │ <span class="green">frame_bytes...</span>
Device → Host: <span class="highlight">0x3E</span> '&gt;' │ <span class="blue">len_lo len_hi</span> │ <span class="green">frame_bytes...</span>
Baud: 115200 Framing: 8N1 Source: mesh/serial.rs</div>
<p>The frame body is a Meshcore <em>Companion</em> command or response. Archipelago builds these in <code>mesh/protocol.rs</code> and parses replies in <code>mesh/listener/decode.rs</code>.</p>
<h3>Companion commands Archipelago uses</h3>
<table>
<thead><tr><th>Code</th><th>Name</th><th>Purpose</th></tr></thead>
<tbody>
<tr><td><code>0x01</code></td><td>APP_START</td><td>Handshake; device returns its node_id and name</td></tr>
<tr><td><code>0x02</code></td><td>SEND_TXT_MSG</td><td>Send payload to a contact (targeted by 6-byte pubkey prefix)</td></tr>
<tr><td><code>0x03</code></td><td>SEND_CHANNEL_TXT_MSG</td><td>Broadcast on a channel (no specific recipient)</td></tr>
<tr><td><code>0x04</code></td><td>GET_CONTACTS</td><td>Pull the device's contact table</td></tr>
<tr><td><code>0x06</code></td><td>SET_DEVICE_TIME</td><td>Sync Unix timestamp for message dating</td></tr>
<tr><td><code>0x07</code></td><td>SEND_SELF_ADVERT</td><td>Broadcast our identity onto the mesh</td></tr>
<tr><td><code>0x08</code></td><td>SET_ADVERT_NAME</td><td>Set our display name</td></tr>
<tr><td><code>0x0A</code></td><td>SYNC_NEXT_MESSAGE</td><td>Pop the next queued inbound message</td></tr>
<tr><td><code>0x0B</code></td><td>SET_RADIO_PARAMS</td><td>Frequency, spreading factor, bandwidth</td></tr>
<tr><td><code>0x0C</code></td><td>SET_RADIO_TX_POWER</td><td>Transmit power (dBm)</td></tr>
<tr><td><code>0x38</code></td><td>GET_STATS</td><td>Device statistics</td></tr>
</tbody>
</table>
<h3>Responses and push notifications</h3>
<p>Responses begin with a status byte. Codes <code>&lt; 0x80</code> are replies to a command we sent; codes <code>&gt;= 0x80</code> are asynchronous push events from the device.</p>
<table>
<thead><tr><th>Code</th><th>Name</th><th>Meaning</th></tr></thead>
<tbody>
<tr><td><code>0x00</code></td><td>RESP_OK</td><td>Command accepted</td></tr>
<tr><td><code>0x01</code></td><td>RESP_ERR</td><td>Command failed + error code</td></tr>
<tr><td><code>0x03</code></td><td>RESP_CONTACT</td><td>One contact entry (32-byte pubkey + metadata)</td></tr>
<tr><td><code>0x05</code></td><td>RESP_SELF_INFO</td><td>Our node_id and name after APP_START</td></tr>
<tr><td><code>0x10</code></td><td>RESP_CONTACT_MSG_V3</td><td>Direct inbound message (SNR + sender prefix + payload)</td></tr>
<tr><td><code>0x11</code></td><td>RESP_CHANNEL_MSG_V3</td><td>Channel broadcast inbound</td></tr>
<tr><td><code>0x83</code></td><td>PUSH_MESSAGES_WAITING</td><td>Async: new messages in queue, call SYNC_NEXT_MESSAGE</td></tr>
</tbody>
</table>
<h2 id="wire">Wire Format — the payload byte 0</h2>
<p>Once a frame reaches the message payload, Archipelago looks at the <strong>first byte</strong> to decide what kind of thing it's dealing with. This single-byte marker is the master switch of the entire mesh protocol.</p>
<div class="diagram"><span class="highlight">0x00</span> Plain text (legacy, unencrypted)
<span class="highlight">0x01</span> Identity broadcast (ARCHY:2 / ARCHY:3)
<span class="highlight">0x02</span> Typed CBOR envelope (plaintext, used for debug or intra-LAN)
<span class="highlight">0xEE</span> Encrypted typed — ChaCha20-Poly1305 w/ static shared secret
<span class="highlight">0xDD</span> Ratcheted typed — Double Ratchet, forward-secure</div>
<p>Markers <code>0xEE</code> and <code>0xDD</code> are the interesting ones — they carry real production traffic. Everything else is either debug or identity bootstrap.</p>
<h3>0xEE — static-key encrypted envelope</h3>
<pre><code>[0xEE] [nonce: 12 bytes] [ciphertext...] [auth tag: 16 bytes]</code></pre>
<ul>
<li>Key: X25519 ECDH between our Ed25519 identity (converted) and the peer's.</li>
<li>Cipher: ChaCha20-Poly1305 AEAD.</li>
<li>Max plaintext: <code>160 − 1 − 12 − 16 = 131</code> bytes (see <code>crypto::MAX_ENCRYPTED_PLAINTEXT</code>).</li>
<li>Properties: confidential + authenticated, <em>but</em> compromise of a key decrypts all history.</li>
</ul>
<h3>0xDD — Double Ratchet envelope</h3>
<pre><code>[0xDD] [RatchetHeader: 40 bytes] [nonce: 12] [ciphertext] [tag: 16]</code></pre>
<ul>
<li>Per-message keys derived via DH ratchet + symmetric-key ratchet (HKDF-SHA256).</li>
<li>Handles out-of-order delivery via a skipped-keys cache.</li>
<li>Properties: forward secrecy + post-compromise recovery. Used for <code>mesh.*</code> chat once a session is established.</li>
<li>Implementation: <code>mesh/ratchet.rs</code>, session load/save in <code>mesh/listener/session.rs</code>.</li>
</ul>
<div class="callout callout-learn">
<strong>Static key vs. ratchet = a safe vs. a self-shredding envelope.</strong>
The <code>0xEE</code> lane is like a locked safe: one key opens everything. The <code>0xDD</code> lane is like handing your friend a new envelope each time, and burning the old one — so even if someone steals next week's key, they can't read last week's messages.
</div>
<h2 id="crypto">Encryption Layers</h2>
<p>Three cryptographic primitives combine to produce the <code>0xDD</code> ratchet flow:</p>
<div class="card-grid">
<div class="card-sm">
<h4>X25519 ECDH</h4>
<p>Each Double Ratchet step generates a fresh keypair. Peers mix the new shared secret into the chain.</p>
</div>
<div class="card-sm">
<h4>HKDF-SHA256</h4>
<p>Derives root key, chain key, and message key at each ratchet step.</p>
</div>
<div class="card-sm">
<h4>ChaCha20-Poly1305</h4>
<p>Symmetric AEAD used for the actual payload encryption + authentication tag.</p>
</div>
</div>
<h3>Session bootstrap — X3DH-like handshake</h3>
<p>Before the ratchet can start, peers exchange a <strong>PrekeyBundle</strong> (type 5) and a <strong>SessionInit</strong> (type 6). Those two messages are carried by the <code>0xEE</code> static-key envelope, because the ratchet session doesn't exist yet. Once <code>SessionInit</code> is processed, subsequent traffic switches to <code>0xDD</code>. See <code>mesh/x3dh.rs</code>.</p>
<h2 id="fragmentation">Fragmentation — how a 500-byte message rides a 160-byte pipe</h2>
<p>The LoRa frame budget is <strong>160 bytes</strong> (<code>protocol::MAX_MESSAGE_LEN</code>). Subtract the marker, nonce, ratchet header, and tag and you end up with ~90 usable plaintext bytes per frame. Anything bigger gets chunked.</p>
<div class="diagram"><span class="highlight">Chunk header</span> ┌──────────┬──────────┬────────────┐
│ type (1) │ id (1) │ total (1) │
└──────────┴──────────┴────────────┘
<span class="highlight">Chunk body</span> Up to 140 bytes of Base64-encoded payload
Sender: compress → encrypt → split into 140-char chunks
→ send with tiny inter-chunk delay
Receiver: accumulate by (sender, chunk_id) → reassemble
→ decrypt → decompress → dispatch</div>
<p>For chat messages shorter than 160 bytes, none of this kicks in — the whole thing fits in one frame. For larger payloads (long messages, forwarded content, PSBTs), the sender splits and the receiver joins.</p>
<div class="callout callout-info">
<strong>Escape hatch: federation fallback.</strong> If a peer is a synthetic federation contact and the message is bigger than 160 bytes, Archipelago <em>skips LoRa entirely</em> and routes the message over Tor federation instead. See the <code>ContentRef</code> path in <code>rpc/mesh/typed_messages.rs</code>.
</div>
<h2 id="dual-transport">Dual Transport — LoRa + Tor federation</h2>
<p>Archipelago treats LoRa and Tor federation as <strong>two lanes of the same highway</strong>. A single chat window may receive some messages over radio and others over onion routing, and the UI doesn't distinguish. The mesh module picks the lane per-message based on the peer type and payload size.</p>
<div class="diagram"> ┌──────────────────┐
│ mesh.send(...) │
└────────┬─────────┘
│
┌──────────┴──────────┐
│ Is peer synthetic? │
└──────────┬──────────┘
No │ Yes
┌──────────┘ └──────────┐
▼ ▼
<span class="highlight">LoRa radio</span> <span class="blue">Tor federation</span>
(160-byte frame) (unlimited, slower setup)
│ │
│ if &gt; 160 B &amp;&amp; synth ──────┘ (fallback)
▼
Chunked over LoRa
or refused if no fallback</div>
<h2 id="addressing">Addressing</h2>
<ul>
<li><strong>Contact ID</strong> — 32-bit handle from Meshcore's contact table. Used by <code>SEND_TXT_MSG</code>.</li>
<li><strong>Pubkey prefix</strong> — first 6 bytes of the peer's Ed25519 public key. Included on the wire so receivers can deduplicate and route replies.</li>
<li><strong>DID / onion</strong> — used for federation peers; synthetic contacts carry the DID so the mesh layer can hand the message to the federation layer.</li>
</ul>
<h2 id="synthetic">Synthetic federation contacts</h2>
<p>To let the chat list show federation peers <em>before</em> any message arrives, Archipelago inserts <strong>synthetic contacts</strong> into the mesh peer list. Their contact IDs live in the upper half of the 32-bit space (<code>≥ 0x8000_0000</code>), derived deterministically from the federation node's Ed25519 pubkey. Collisions with real LoRa contact IDs are impossible by construction.</p>
<hr>
<h2 id="msg-overview">All 23 Message Types</h2>
<p>Every typed message is a CBOR envelope identified by a single <code>MeshMessageType</code> byte. The <strong>Transport</strong> column shows which marker carries it on the wire and which Companion command is used.</p>
<table>
<thead><tr>
<th>ID</th><th>Type</th><th>Purpose</th><th>Marker</th><th>Cmd</th><th>Chunked?</th>
</tr></thead>
<tbody>
<tr><td>0</td><td>Text</td><td>Plain chat message</td><td>0xDD</td><td>0x02</td><td>If &gt;160 B</td></tr>
<tr><td>1</td><td>Alert</td><td>Emergency / dead-man heartbeat</td><td>0xDD</td><td>0x02/0x03</td><td>No (short)</td></tr>
<tr><td>2</td><td>Invoice</td><td>Lightning / BOLT11 invoice</td><td>0xDD</td><td>0x02</td><td>Usually</td></tr>
<tr><td>3</td><td>PsbtHash</td><td>Unsigned tx hash for co-signing</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>4</td><td>Coordinate</td><td>GPS location share</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>5</td><td>PrekeyBundle</td><td>X3DH bootstrap (pre-session)</td><td>0xEE</td><td>0x02</td><td>No</td></tr>
<tr><td>6</td><td>SessionInit</td><td>Initial ratchet message</td><td>0xEE</td><td>0x02</td><td>No</td></tr>
<tr><td>7</td><td>BlockHeader</td><td>Bitcoin block height/hash</td><td>0xDD</td><td>0x03</td><td>No</td></tr>
<tr><td>8</td><td>TxRelay</td><td>Signed Bitcoin tx for on-grid peer to broadcast</td><td>0xDD</td><td>0x02</td><td>Yes</td></tr>
<tr><td>9</td><td>TxRelayResponse</td><td>txid or error from the relay peer</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>10</td><td>LightningRelay</td><td>BOLT11 to pay via on-grid peer</td><td>0xDD</td><td>0x02</td><td>Yes</td></tr>
<tr><td>11</td><td>LightningRelayResponse</td><td>payment_hash or error</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>12</td><td>TxConfirmation</td><td>Depth update (1/2/3 confs)</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>13</td><td>Reply</td><td>Quoted reply to a previous message</td><td>0xDD</td><td>0x02</td><td>If long</td></tr>
<tr><td>14</td><td>Reaction</td><td>Emoji reaction on MessageKey</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>15</td><td>ReadReceipt</td><td>"Seen up to MessageKey X"</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>16</td><td>Forward</td><td>Re-forwarded original w/ provenance</td><td>0xDD</td><td>0x02</td><td>Yes</td></tr>
<tr><td>17</td><td>Edit</td><td>In-place text replacement</td><td>0xDD</td><td>0x02</td><td>If long</td></tr>
<tr><td>18</td><td>Delete</td><td>Tombstone for earlier message</td><td>0xDD</td><td>0x02</td><td>No</td></tr>
<tr><td>19</td><td>ContentRef</td><td>CID of blob held by sender (file/image)</td><td>0xDD</td><td>0x02 or Tor</td><td>Federation fallback</td></tr>
<tr><td>20</td><td>Presence</td><td>Heartbeat + last-activity epoch</td><td>0xDD</td><td>0x03</td><td>No</td></tr>
<tr><td>21</td><td>ChannelInvite</td><td>Group membership announcement</td><td>0xDD</td><td>0x03</td><td>No</td></tr>
<tr><td>22</td><td>ContactCard</td><td>Shareable federation node card</td><td>0xDD</td><td>0x02</td><td>Maybe</td></tr>
</tbody>
</table>
<p>The remaining sections walk through each category and explain both the sender-side code path and what the bytes look like on the air.</p>
<h2 id="msg-text">Text, Reply, Edit, Delete, Forward</h2>
<h3>Text (type 0)</h3>
<p><strong>Sender path.</strong> <code>rpc.mesh.send</code> → <code>typed_messages::send_text</code> → CBOR-encode the <code>Text{body}</code> variant → ratchet-encrypt → prefix <code>0xDD</code> → if under 160 B, send in one <code>SEND_TXT_MSG</code> frame; otherwise split into Base64 chunks and send sequentially with a small inter-frame sleep so the radio doesn't overflow its TX buffer.</p>
<h3>Reply (type 13)</h3>
<p>Same as Text, but the CBOR envelope carries a <code>MessageKey</code> pointing at the parent message (sender pubkey prefix + timestamp). The UI renders a quote banner; the wire cost is ~12 extra bytes.</p>
<h3>Edit (type 17)</h3>
<p>Envelope contains the original <code>MessageKey</code> plus the new body. Receiver updates its local store in-place and tags the entry "edited".</p>
<h3>Delete (type 18)</h3>
<p>Tombstone only: <code>MessageKey</code> with no body. Receivers keep the original bytes but mark the row deleted. Costs ~20 bytes on the wire.</p>
<h3>Forward (type 16)</h3>
<p>Wraps original <code>{sender_name, original_timestamp, body}</code> so the receiver can render "Forwarded from &lt;name&gt;". Because the body is nested, forwards are <em>almost always</em> chunked.</p>
<h2 id="msg-social">Reaction, ReadReceipt, Presence</h2>
<h3>Reaction (type 14)</h3>
<p>Envelope: <code>{target: MessageKey, emoji: String}</code>. Single-frame, single-emoji. Receiver aggregates reactions per MessageKey and shows them as inline chips (see <code>MessageActions</code> in <code>neode-ui</code>).</p>
<h3>ReadReceipt (type 15)</h3>
<p>Envelope: <code>{up_to: MessageKey}</code>. Semantically "I've seen everything up to and including this message." One receipt covers all prior unread, so traffic is O(1) per read burst rather than O(n).</p>
<h3>Presence (type 20)</h3>
<p>Periodic heartbeat carrying <code>{last_activity_epoch}</code>. Broadcast on a channel (<code>SEND_CHANNEL_TXT_MSG</code>, cmd <code>0x03</code>) rather than to a specific peer, so every listener updates their "last seen" indicator in one shot.</p>
<div class="callout callout-learn">
<strong>Like a lighthouse beacon.</strong> Presence doesn't go to anyone in particular — it's a flash that everyone in radio range can see. "I'm still here, last active two minutes ago." Cheap and unaddressed.
</div>
<h2 id="msg-content">ContentRef — files and images without bloating the radio</h2>
<p>LoRa cannot move a 500 KB image. The <code>ContentRef</code> type (19) solves this by sending only a <strong>pointer</strong> — a content ID (CID) plus a tiny thumbnail or description — and letting the receiver fetch the full blob out-of-band over Tor federation.</p>
<div class="diagram">Sender Receiver
────── ────────
store blob locally (CID)
┌──────────────────────┐
│ ContentRef {cid, │ ──ratchet──▶
│ mime, size, │ 0xDD
│ thumb_hash} │ over LoRa
└──────────────────────┘
see CID in chat
click to fetch
┌─────────────────┐
│ rpc.mesh.fetch- │
│ content(cid) │
└────────┬────────┘
▼
federation (Tor)
resolve DID → pull blob</div>
<div class="callout callout-info">
<strong>Resolution bug fix note.</strong> An earlier revision of <code>ContentRef</code> routed the fetch via a name-match on the contact list, which broke when two peers had the same display name. The fix (see commit <code>5f7ebf14</code>) resolves the owning peer by DID and falls back to name-match only if DID lookup fails.
</div>
<h2 id="msg-bitcoin">Bitcoin &amp; Lightning over LoRa</h2>
<p>Archipelago uses the mesh as a <strong>Bitcoin transport of last resort</strong>. Signed transactions travel from an offline signer, through the mesh, to a peer with internet, who then rebroadcasts them to the Bitcoin network and reports back.</p>
<h3>TxRelay (8) → TxRelayResponse (9) → TxConfirmation (12)</h3>
<div class="diagram">Offline signer On-grid relay peer Bitcoin p2p
────────────── ────────────────── ───────────
sign tx
┌─────────────┐
│ TxRelay │ ─ratchet/LoRa▶ decrypt → validate
│ {raw_tx} │ broadcast via bitcoind ───▶ mempool
└─────────────┘ │
▼
┌────────────────────────┐
◀─ratchet│ TxRelayResponse{txid} │
└────────────────────────┘
(or {error})
later, as blocks arrive:
┌────────────────────────┐
◀─ratchet│ TxConfirmation │
│ {txid, depth: 1..3} │
└────────────────────────┘</div>
<p>The binary framing in <code>mesh/bitcoin_relay.rs</code> is intentionally tight — raw binary, not CBOR — to keep a signed 1-input/1-output tx inside one or two 160-byte frames. Confirmation updates are tiny (txid + depth byte) and ride in a single frame.</p>
<h3>LightningRelay (10) → LightningRelayResponse (11)</h3>
<p>Same shape but the payload is a BOLT11 invoice string. The relay peer pays the invoice from its own node and returns <code>payment_hash</code> or an error. Invoices are often long enough to chunk.</p>
<h3>Invoice (2) and PsbtHash (3)</h3>
<p>These are <em>not</em> relays — they're peer-to-peer handoffs. <code>Invoice</code> delivers a BOLT11 to be paid by the recipient. <code>PsbtHash</code> carries just the hash of an unsigned PSBT so the recipient can retrieve the full PSBT out-of-band and co-sign.</p>
<h3>BlockHeader (7)</h3>
<p>Off-grid nodes need a recent block height to avoid being fooled by stale data. A BlockHeader broadcast (sent via <code>SEND_CHANNEL_TXT_MSG</code>) lets anyone in range learn the latest height and hash from any peer with internet. Tiny payload: 4 bytes height + 32 bytes hash.</p>
<h2 id="msg-safety">Alerts, Coordinates, Dead-Man</h2>
<h3>Alert (type 1)</h3>
<p>Envelope: <code>{kind, message, sender_contact_id}</code>. Kinds include <code>Emergency</code> and <code>Deadman</code>. Alerts can be sent direct-to-contact (for family) or channel-broadcast (for community).</p>
<h3>Dead-man switch</h3>
<p>A background task in <code>mesh/alerts.rs</code> sends a <code>Deadman</code> alert on a configurable interval (default 6 hours). If the user doesn't touch the UI within that window, the alert fires automatically and asks chosen recipients to check in. Powered off? The next peer to receive your last heartbeat notices the gap.</p>
<h3>Coordinate (type 4)</h3>
<p>Envelope: <code>{lat, lon, accuracy_m}</code> with lat/lon as fixed-point integers to stay under 16 bytes. Used for off-grid location sharing — hiking, sailing, field ops.</p>
<h3>ChannelInvite (type 21)</h3>
<p>Phase 5 group chat primitive. Announces a new channel and its membership so other nodes can subscribe. Broadcast via <code>SEND_CHANNEL_TXT_MSG</code>.</p>
<h2 id="msg-identity">Identity, PrekeyBundle, ContactCard</h2>
<h3>Identity broadcast (marker 0x01, ARCHY:2/3)</h3>
<p>The handshake. Before any ratchet session exists, a node advertises its Ed25519 public key on the mesh with an identity packet prefixed <code>0x01</code>. This is how peers discover each other. The payload encodes protocol version (<code>ARCHY:2</code> or <code>ARCHY:3</code>) and the raw pubkey. Carried by <code>CMD_SEND_SELF_ADVERT</code> (<code>0x07</code>).</p>
<h3>PrekeyBundle (type 5) and SessionInit (type 6)</h3>
<p>X3DH handshake. <code>PrekeyBundle</code> advertises a signed prekey; <code>SessionInit</code> consumes it to derive the initial ratchet root key. Both ride on <code>0xEE</code> (static-key encryption), because the ratchet session they're creating doesn't yet exist.</p>
<h3>ContactCard (type 22)</h3>
<p>A shareable card containing <code>{did, onion_address, pubkey, display_name}</code>. When a receiver taps "add" on the card, Archipelago one-click federates with that node over Tor. This is the bridge that lets LoRa-discovered peers become full federation contacts.</p>
<hr>
<h2 id="rpc">RPC API — what callers actually invoke</h2>
<p>Every user-facing action goes through the RPC dispatcher (<code>api/rpc/dispatcher.rs</code>, lines 287+) and ends in <code>api/rpc/mesh/typed_messages.rs</code>. The tables below show the public surface.</p>
<h3>Core commands</h3>
<table>
<thead><tr><th>RPC</th><th>Effect</th></tr></thead>
<tbody>
<tr><td><code>mesh.status</code></td><td>Device info, peer count, enabled state</td></tr>
<tr><td><code>mesh.peers</code></td><td>List all discovered peers with RSSI / SNR / hop count</td></tr>
<tr><td><code>mesh.messages</code></td><td>Retrieve stored mesh messages</td></tr>
<tr><td><code>mesh.send</code></td><td>Send plain text to a specific peer</td></tr>
<tr><td><code>mesh.send-channel</code></td><td>Broadcast on a channel</td></tr>
<tr><td><code>mesh.broadcast</code></td><td>Mesh-wide announcement</td></tr>
<tr><td><code>mesh.configure</code></td><td>Set device params (name, power, channel)</td></tr>
<tr><td><code>mesh.debug-dump</code></td><td>Raw state for debugging</td></tr>
</tbody>
</table>
<h3>Rich message commands</h3>
<table>
<thead><tr><th>RPC</th><th>Msg Type</th><th>Notes</th></tr></thead>
<tbody>
<tr><td><code>mesh.send-invoice</code></td><td>Invoice (2)</td><td>Deliver BOLT11 to peer</td></tr>
<tr><td><code>mesh.send-coordinate</code></td><td>Coordinate (4)</td><td>Single frame, fixed-point</td></tr>
<tr><td><code>mesh.send-alert</code></td><td>Alert (1)</td><td>Emergency or deadman</td></tr>
<tr><td><code>mesh.send-content</code></td><td>ContentRef (19)</td><td>Stores blob, sends CID</td></tr>
<tr><td><code>mesh.fetch-content</code></td><td>—</td><td>Pulls blob via federation</td></tr>
<tr><td><code>mesh.send-psbt</code></td><td>PsbtHash (3)</td><td>Hash only, full PSBT via fetch</td></tr>
<tr><td><code>mesh.send-reply</code></td><td>Reply (13)</td><td>Quoted response</td></tr>
<tr><td><code>mesh.send-reaction</code></td><td>Reaction (14)</td><td>Emoji</td></tr>
<tr><td><code>mesh.send-read-receipt</code></td><td>ReadReceipt (15)</td><td>Cumulative "seen up to"</td></tr>
<tr><td><code>mesh.forward-message</code></td><td>Forward (16)</td><td>Wraps original + provenance</td></tr>
<tr><td><code>mesh.edit-message</code></td><td>Edit (17)</td><td>In-place text replacement</td></tr>
<tr><td><code>mesh.delete-message</code></td><td>Delete (18)</td><td>Tombstone</td></tr>
</tbody>
</table>
<h2 id="ui">User Interface</h2>
<p>The Vue side lives under <code>neode-ui/src/views/mesh/</code> with state in <code>stores/mesh.ts</code>. Notable panels:</p>
<div class="card-grid">
<div class="card-sm">
<h4>Mesh chat</h4>
<p>Telegram-style UI with reply banners, inline reaction chips, forward/edit/delete action menu, read-receipts, outbox status.</p>
</div>
<div class="card-sm">
<h4>MeshBitcoinPanel</h4>
<p>UI for TxRelay / LightningRelay submission and confirmation tracking.</p>
</div>
<div class="card-sm">
<h4>MeshDeadmanPanel</h4>
<p>Configure dead-man interval, pick recipients, show last heartbeat time.</p>
</div>
<div class="card-sm">
<h4>Unified inbox</h4>
<p>Federation and mesh chats appear side-by-side; the transport is invisible to the user.</p>
</div>
</div>
<h2 id="listener">Listener loop — how inbound traffic is decoded</h2>
<p>A long-running async task in <code>mesh/listener/mod.rs</code> owns the serial device and feeds events into the rest of the system.</p>
<div class="diagram">loop {
event = await serial_read()
match event {
<span class="green">PUSH_MESSAGES_WAITING</span> → send SYNC_NEXT_MESSAGE until empty
<span class="green">RESP_CONTACT_MSG_V3</span> → decode.rs extracts payload
→ match first byte:
<span class="highlight">0x00</span> plain text
<span class="highlight">0x01</span> identity → frames::parse_identity
<span class="highlight">0x02</span> typed CBOR plaintext
<span class="highlight">0xEE</span> → crypto::decrypt_static
<span class="highlight">0xDD</span> → session::load + ratchet::decrypt
→ dispatch.rs routes typed msg
to chat store / bitcoin relay /
alerts / presence / ...
<span class="green">RESP_CONTACT</span> → contact list update
<span class="green">RESP_SELF_INFO</span> → record our node_id
}
}</div>
<p>Chunk reassembly happens in <code>listener/session.rs</code>, keyed by <code>(sender_pubkey_prefix, chunk_id)</code>. Incomplete chunks expire after a timeout so a lost frame doesn't leak memory.</p>
<h2 id="files">File Map</h2>
<table>
<thead><tr><th>File</th><th>Size</th><th>Role</th></tr></thead>
<tbody>
<tr><td><code>mesh/mod.rs</code></td><td>52 KB</td><td>Public API, send paths, federation integration</td></tr>
<tr><td><code>mesh/protocol.rs</code></td><td>26 KB</td><td>Frame encoding/decoding, command builders</td></tr>
<tr><td><code>mesh/serial.rs</code></td><td>15 KB</td><td>USB driver, device detection, handshake</td></tr>
<tr><td><code>mesh/crypto.rs</code></td><td>10 KB</td><td>X25519 ECDH, ChaCha20-Poly1305, HKDF</td></tr>
<tr><td><code>mesh/ratchet.rs</code></td><td>16 KB</td><td>Double Ratchet implementation</td></tr>
<tr><td><code>mesh/message_types.rs</code></td><td>23 KB</td><td>23 typed message discriminators + CBOR schemas</td></tr>
<tr><td><code>mesh/bitcoin_relay.rs</code></td><td>17 KB</td><td>TxRelay / LightningRelay binary framing</td></tr>
<tr><td><code>mesh/listener/dispatch.rs</code></td><td>29 KB</td><td>Typed-message routing into chat/relay/alerts</td></tr>
<tr><td><code>mesh/listener/session.rs</code></td><td>14 KB</td><td>Ratchet session persistence + chunk reassembly</td></tr>
<tr><td><code>mesh/x3dh.rs</code></td><td>—</td><td>Prekey / SessionInit bootstrap</td></tr>
<tr><td><code>mesh/outbox.rs</code></td><td>—</td><td>Retry queue for unacked sends</td></tr>
<tr><td><code>mesh/steganography.rs</code></td><td>—</td><td>Weather/sensor framing for deniable traffic</td></tr>
<tr><td><code>api/rpc/mesh/typed_messages.rs</code></td><td>—</td><td>All <code>mesh.*</code> RPC handlers</td></tr>
<tr><td><code>neode-ui/src/stores/mesh.ts</code></td><td>14 KB</td><td>Pinia store consumed by all mesh Vue views</td></tr>
</tbody>
</table>
<hr>
<h2>Summary scoreboard</h2>
<div class="score-grid">
<div class="score-card"><div class="score">23</div><div class="label">Message types</div></div>
<div class="score-card"><div class="score">160</div><div class="label">Bytes / frame</div></div>
<div class="score-card"><div class="score">2</div><div class="label">Transports</div></div>
<div class="score-card"><div class="score">5</div><div class="label">Wire markers</div></div>
<div class="score-card"><div class="score">~6k</div><div class="label">LoC in mesh/</div></div>
<div class="score-card"><div class="score">FS</div><div class="label">Forward-secure</div></div>
</div>
<div class="callout callout-success">
<strong>Bottom line.</strong> Archipelago's mesh isn't a chat toy. It's a complete off-grid transport with forward-secure end-to-end encryption, 23 typed message kinds, Bitcoin and Lightning relay, fragmentation, store-and-forward, and a seamless Tor federation fallback. From the user's perspective it looks like iMessage; from the wire's perspective it's a carefully budgeted 160 bytes of ChaCha20 ciphertext riding on a sub-kbps radio link.
</div>
</main>
</body>
</html>
docs/bitcoin-multi-version-design.md
+4 -93
View File
@@ -1,91 +1,7 @@
# Bitcoin Multi-Version Support — Design
<!-- ════════════════════════════════════════════════════════════════════
PROGRESS TRACKER / RESUME POINT (keep this current — update each session)
════════════════════════════════════════════════════════════════════
**Branch/worktree:** `bitcoin-multi-version` @ `/home/archipelago/Projects/archy-btcver`
(isolated — never touch `main` or the other agent's branch). All work UNCOMMITTED on
that branch as of last update.
**Last updated:** 2026-06-28 (session 2 — software end-to-end implemented)
**Motivation refresh:** BIP-110 signalling makes per-node version *choice* a real
requirement — runners must be able to pick / pin / switch Core & Knots versions.
**User direction this session:** finish the SOFTWARE end-to-end (Phase 1–3 + UI),
DEFER the Phase 0 image build pipeline. Downgrade policy = **warn + confirm + allow**.
### Status by phase
- [x] **Phase 1 — catalog schema** (`app_catalog.rs`): `CatalogVersion` struct +
`versions[]` + `catalog_versions()` / `catalog_default_version()` /
`catalog_image_for_version()` (same-repo guard) DONE. Pin suppresses update badge
in `available_update_for_app()` DONE. `versions[]` now EMITTED by
`scripts/generate-app-catalog.sh` (curated `VERSIONS` map) → `releases/app-catalog.json`
regenerated; bitcoin-core carries its one built version (28.4.0, default). **Knots
versions[] intentionally empty** (only floating `:latest` exists; design forbids
advertising floating). More versions light up automatically once Phase 0 builds
tagged images and they're appended to the `VERSIONS` map.
- [x] **Phase 2 — install-time selection**: `version_config.rs` (pin/auto-update
persistence + `is_downgrade()` + `auto_update_apps()`, unit-tested) DONE;
`install.rs` `persist_install_version_selection()` DONE; `prod_orchestrator.rs`
pinned-wins resolution DONE. **UI:** `MarketplaceAppDetails.vue` install panel shows
a version `<select>` (latest pre-selected) when the app offers ≥2 versions — passes
the choice to `package.install`. (Hidden today since only 1 version exists.)
- [x] **Phase 3 — in-app switch + auto-update toggle**:
- `package.versions` RPC (read) + `package.set-config` RPC (write, downgrade-gated)
→ new `api/rpc/package/set_config.rs`, wired in `mod.rs` + `dispatcher.rs`.
- Auto-update tick: `run_update_scheduler` now takes the orchestrator + calls
`apply_per_app_auto_updates()` hourly (opt-in, pin-respecting, catalog-driven).
- UI: "Version & Updates" card in `appDetails/AppSidebar.vue` (version switch +
auto-update toggle + downgrade warn/confirm); `rpc-client.ts` + types added.
- [x] **Phase 0 — image build pipeline**: `scripts/build-bitcoin-image.sh` —
downloads the OFFICIAL upstream tarball + SHA256SUMS(.asc), verifies SHA-256 **and**
the OpenPGP signature (fail-closed; pinned release-key fingerprints), builds a
minimal **rootless** image (debian-slim + verified `bitcoind`/`bitcoin-cli`),
smoke-tests `--version`, tags + pushes `:<version>`. Validated on Core 31.0
(pinned-GPG pass, smoke `v31.0.0`). **Published curated set** (registry
`lfg2025`): Core **31.0, 30.2, 29.3, 27.2, 26.2, 25.2** (28.4 already present —
kept, not overwritten) + Knots **29.3.knots20260508**. `VERSIONS` map in
`generate-app-catalog.sh` lists them; catalog regenerated. Adding a future release
= run the script for it, then prepend it to the map + regenerate.
### Verification status
- `cargo check -p archipelago` GREEN (backend). Frontend `npm run build` GREEN
(vue-tsc typecheck passes; new RPC strings confirmed in `web/dist`).
- Unit tests: `version_config` had a pre-existing parallel-test race (shared
process-global `ARCHIPELAGO_DATA_DIR`) — FIXED with an `ENV_LOCK` mutex + unique
per-test dirs. `set_config` `image_tag` test added.
- **Phase 0 images verified end-to-end**: SHA-256 + pinned-maintainer OpenPGP
signature (deterministic VALIDSIG check), built rootless, smoke-tested, **pushed
to the live registry** — confirmed remotely: `bitcoin` tags
{25.2,26.2,27.2,28.4,29.3,30.2,31.0} + `bitcoin-knots:29.3.knots20260508`.
- **NOT yet verified on `.228`** (CLAUDE.md invariant — do before any tag): install
bitcoin-core, open its page, switch/pin a version, confirm recreate. All code
UNCOMMITTED on the branch.
### Gotchas captured (for resume)
- `gpg --verify` exit code is unreliable on multi-sig `SHA256SUMS` — must parse
`--status-fd` VALIDSIG and require a pinned maintainer fpr (script does this).
- `podman push` needs the sandbox disabled (`/var/tmp` is RO under the harness
sandbox) and `--tls-verify=false` (registry serves HTTP). Persistent keyring
(`BITCOIN_KEYRING_DIR`) avoids flaky per-build keyserver fetches.
### Next action when resuming
1. Re-verify: `cd archy-btcver/core && CARGO_INCREMENTAL=0 cargo check -p archipelago`
and `cargo test -p archipelago -- version_config set_config`; `cd neode-ui && npm run build`.
2. Live-verify on `.228`: install bitcoin-core, open its detail page → "Version &
Updates" card; exercise `package.versions` / `package.set-config` via RPC.
3. Commit on the branch (checkpoint).
4. **Phase 0** when greenlit: build+push tagged Core/Knots images, then extend the
`VERSIONS` map in `scripts/generate-app-catalog.sh` and regenerate the catalog.
### Decisions still needed from user (see §6 open questions)
Curated version set + storage budget (defaulted to current+~3 majors); when to do
Phase 0 image pipeline; pruned-node downgrade policy refinement (currently warn+confirm
for all). Auto-update default = OFF (opt-in), as recommended.
════════════════════════════════════════════════════════════════════ -->
**Status:** design (2026-06-22)
**Status:** implemented — all four phases shipped (catalog schema, install-time selection, in-app switch + auto-update toggle, and the verified image build pipeline). Downgrades are guarded: the update path never offers a lower version than what is running.
**Goal:** let a user choose *which* version of Bitcoin Core / Bitcoin Knots to
install (latest pre-selected, older versions in a dropdown), and later switch
versions or opt into auto-update — all manifest/catalog-driven, all served from
@@ -94,13 +10,8 @@ changes.
See also: [`docs/registry-manifest-design.md`](registry-manifest-design.md)
(catalog distribution + signing this builds on),
[`docs/PRODUCTION-MASTER-PLAN.md`](PRODUCTION-MASTER-PLAN.md) (gate that must be
green first), `MEMORY → project_decoupled_app_updates`,
`MEMORY → project_manifest_driven_north_star`.
and the production test gate (which must be green first).
> **Scheduling:** this is net-new scope. It lands **after** the production test
> gate (`tests/lifecycle/run-20x.sh`) is green on `.228` + `.198`. The data-
> preservation invariant (downgrade vs. chainstate) is the highest risk here.
---
@@ -111,7 +22,7 @@ green first), `MEMORY → project_decoupled_app_updates`,
|-------|-------|
| `apps/bitcoin-core/Dockerfile` | `FROM bitcoin/bitcoin:24.0` — a **community** image, **stale** (manifest says 28.4), no project-official Docker image exists |
| `apps/bitcoin-knots/` | **no Dockerfile** — `:latest` is built/pushed by hand |
| Registry | `scripts/image-versions.sh` → `ARCHY_REGISTRY="146.59.87.168:3000/lfg2025"`; only `BITCOIN_KNOTS_IMAGE=…/bitcoin-knots:latest` pinned, no Core pin |
| Registry | `scripts/image-versions.sh` → `ARCHY_REGISTRY="source.archipelago-foundation.org/lfg2025"`; only `BITCOIN_KNOTS_IMAGE=…/bitcoin-knots:latest` pinned, no Core pin |
| Tags in registry | **one tag per image**. No historical versions. |
### Version pinning
@@ -260,7 +171,7 @@ free. `version`/`image` stay as the default for back-compat.
- **Floating tags** (`latest`) are never advertised as a selectable "version" and
never counted as an available update (already handled by
`available_update_for_app`).
- **Verify on a real node** (`.228` then `.198`) and pass `run-20x` before any
- **Verify on a real node** and pass the lifecycle gate before any
tag.
---
docs/bitcoin-rpc-relay.md
+2 -2
View File
@@ -171,7 +171,7 @@ http://<archipelago-lan-ip>:80
For the tested node the LAN upstream was:
```text
http://192.168.1.116:80
http://archipelago.local:80
```
The public proxy should serve a valid TLS certificate for the chosen subdomain.
@@ -276,7 +276,7 @@ Expected result:
The working endpoint used in this setup was:
```text
https://shard.tx1138.com/
https://<your-mempool-instance>/
```
It was verified with:
docs/bitcoin-version-bulletproof-rollout.md
-131
View File
@@ -1,131 +0,0 @@
# Bitcoin Multi-Version — Bulletproofing & Rollout (handoff)
> **Status 2026-06-29:** code + images + catalog + frontend DONE on branch
> `bitcoin-version-bulletproof` (base commit `095a76cd`, plus the catalog-generator
> + handoff follow-ups). **.228 is the test node**: binary + frontend + catalog are
> live there; its Knots chainstate is mid-**reindex recovery** (see §5). The fleet
> rollout (OTA binary+frontend, mirror catalog publish, `:latest` repoint) is the
> **coordinated step the other agent owns** — see §4. Pairs with
> `docs/bitcoin-multi-version-design.md` (the original design).
## 1. What was broken (root causes)
User report: "switched Knots to `v29.3.knots20260508`, version didn't update in the UI."
Three **stacked** bugs, plus a data-corruption hazard:
1. **Reconciler reverted the pin.** `prod_orchestrator::sync_quadlet_unit` re-rendered the
quadlet every reconcile tick using the manifest's `:latest`, ignoring the per-app
pinned version → any switch silently reverted within one tick.
2. **Entrypoint render bug.** The renderer folded the manifest `entrypoint: ["sh","-lc"]`
into `Exec=`. That only works when the image ENTRYPOINT is a passthrough shell wrapper.
The versioned images use `ENTRYPOINT ["bitcoind"]`, so `Exec=sh -lc …` became
`bitcoind sh -lc …` → `unexpected token 'sh'` → crash loop.
3. **Image USER divergence.** The versioned images were built `USER bitcoin` (uid 1000);
the legacy `:latest` ran as **root**. Chain data is owned by the `data_uid`
(host 100101 / container uid 102). Root reads it via `CAP_DAC_OVERRIDE` (granted in the
manifest); uid-1000 cannot → `Error initializing block database`.
4. **Data hazard (already hit on .228).** Repeated failed starts under mixed UIDs left
bitcoind's two LevelDBs (`blocks/index/` + `chainstate/`) truncated to KB stubs while
the raw `blocks/blk*.dat` (797 GB) stayed intact. Recovery = `bitcoind -reindex` from
local blocks (no re-download). The uniform-root image fix (below) removes the mixed-UID
cause going forward; the proper switch flow was already data-safe (600s stop grace,
clean stop→rm→recreate, conflict-stops the other impl — they share port 8332 + datadir
`/var/lib/archipelago/bitcoin`).
## 2. What was fixed (all on the branch)
- **Renderer** (`core/archipelago/src/container/`):
- `prod_orchestrator.rs`: factored `resolve_catalog_image()` (catalog/pinned-version →
image) and call it in BOTH `install_fresh` and `sync_quadlet_unit` — the pin now
survives reconcile.
- `quadlet.rs`: emit a real `Entrypoint=<first>` + `Exec=<rest+cmd>` instead of folding;
`exec_changed` now also diffs `Entrypoint=` so the recreate fires. Validated against
the live podman 5.4.2 quadlet generator.
- **Images** (`scripts/build-bitcoin-image.sh`, `apps/bitcoin-{knots,core}/Dockerfile`):
removed `USER bitcoin` → run as **container-root** like legacy (still 100% rootless:
container-root maps to the unprivileged host service user; `CAP_DAC_OVERRIDE` from the
manifest lets bitcoind read the `data_uid`-owned datadir). **All** images rebuilt root +
pushed to the mirror (`146.59.87.168:3000/lfg2025`):
- Knots: `29.3.knots20260508`, `29.3.knots20260507`, `29.3.knots20260210`, `29.2.knots20251110`
- Core: `25.2 26.2 27.2 28.4 29.2 29.3 30.2 31.0` + `latest` (→31.0)
- **Catalog** (`scripts/generate-app-catalog.sh` VERSIONS map + regenerated
`releases/app-catalog.json`): Knots & Core `versions[]` populated; the generator now
forces top-level `version` == the `default` entry's version (the `169ff2e2` invariant)
regardless of the manifest version. Knots `latest` entry points at the newest **dated**
image (`29.3.knots20260508`) so "Always use latest" = newest on fixed-binary nodes.
- **Frontend** (`neode-ui/`):
- `AppSidebar.vue`: rename the latest option to **"Always use the latest version"**
(no `v` prefix), fix right padding, and `pickSelection()` guarantees the bound value is
a real option (fixes the blank dropdown).
- New `components/InstallVersionModal.vue`: full-screen version chooser shown from the
App Store / Discover **card** install button for multi-version apps — app icon +
"Install <name>", latest pre-selected. Wired in `Discover.vue handleInstall`.
- i18n keys: `appDetails.alwaysUseLatestVersion`, `marketplace.installModalTitle/Hint`.
## 3. Current live state on .228 (test node)
- Binary with both renderer fixes: **deployed** (`/usr/local/bin/archipelago`).
- New frontend bundle: **deployed** to `/opt/archipelago/web-ui` (hard-refresh to see it).
- Updated catalog: placed at `/var/lib/archipelago/app-catalog.json` (local override —
will refresh from the mirror's OLDER copy at the next hourly fetch until §4 publishes it).
- Knots: `bitcoin-knots` service held **stopped** (`package.stop`, user_stopped);
a detached `bitcoin-knots-reindex` container is rebuilding the index+UTXO (§5).
## 4. Remaining — coordinated fleet rollout (OTHER AGENT)
Do this together with the other workstream's release, AFTER both are ready:
1. **Merge** branch `bitcoin-version-bulletproof` into the release line.
2. **Build + OTA** the binary + frontend (these carry the renderer fix + UI). The renderer
fix is a **hard prerequisite** for the new images everywhere — see fleet-safety below.
3. **Publish the catalog** to the mirror (push `releases/app-catalog.json` to gitea-vps2
`main`, the raw URL nodes fetch hourly). The current catalog is **fleet-safe even before
the binary lands**: unpinned/auto-update nodes resolve via the manifest's floating
`:latest` (still the legacy image); only explicit version selection (needs the new UI)
uses the new root images.
4. **Only AFTER the binary is fleet-wide:** optionally repoint the `bitcoin-knots:latest`
tag → `29.3.knots20260508` (root) and simplify the catalog `latest` entry back to the
`:latest` tag. **Do NOT repoint `:latest` before then** — old-binary nodes fold
`Exec=sh -lc …` and would crash on an `ENTRYPOINT ["bitcoind"]` image. (Core never
worked on old binaries — it always shipped `ENTRYPOINT ["bitcoind"]` — so Core has no
such constraint.)
5. **Verify the full switch matrix** on a healthy node (§6).
## 5. Finishing .228's reindex (OTHER AGENT owns this — not babysat by the original author)
The detached `bitcoin-knots-reindex` container runs the new **root** `29.3.knots20260508`
image with `-reindex -server=0` against `/var/lib/archipelago/bitcoin`. It holds the datadir
lock, so the managed service (held stopped) can't collide. When it has connected blocks up
to ~the prior tip (height ≥ ~955800) it's done; then:
```sh
# on .228 (SSH/sudo/UI pw all: ThisIsWeb54321@)
podman stop -t 600 bitcoin-knots-reindex && podman rm bitcoin-knots-reindex
# start the managed service via RPC (sets desired=running, clears user_stopped):
# package.start {id: bitcoin-knots} (POST https://127.0.0.1/rpc/v1, CSRF: echo csrf_token cookie as X-CSRF-Token)
# verify:
podman exec bitcoin-knots sh -lc '$(command -v bitcoind) --version | head -1' # → v29.3.knots20260508
# RPC up → the Bitcoin UI populates; it syncs the gap to tip.
```
The "Bitcoin RPC connection refused (127.0.0.1:8332)" the UI shows is EXPECTED until this
swap (reindex runs with RPC off).
## 6. Switch-matrix test plan (what "bulletproof" must prove)
On a healthy node, each step must end with bitcoind running + RPC answering + syncing, with
NO `Error initializing block database` and NO data loss:
- Knots: switch `latest` → `29.3.knots20260507` → `29.3.knots20260210` → back to `latest`.
- Core: install `latest`; switch `31.0` → `28.4.0`.
- **Knots ↔ Core** (shared datadir/port): Knots→Core upgrade path (Core ≥ data version) and
the reverse. **Cross-major DOWNGRADES** (e.g. 29.x data → Core 28.4) legitimately need a
reindex — the UI already surfaces a downgrade warning; confirm it does and that confirming
reindexes cleanly rather than crash-looping.
- Reboot survival after each switch.
## 7. Notes / assumptions
- **"29.2"** in the request doesn't exist as a Knots build (404 upstream); added as **Bitcoin
Core 29.2** (exists). Revisit if a Knots 29.2 was meant.
- Reindex is unavoidable ONLY because .228's index was already corrupted by the pre-fix
crash loop; a normal switch on the fixed binary does NOT reindex.
- Creds for .228: SSH/sudo + UI/RPC all `ThisIsWeb54321@`.
docs/bulletproof-containers.md
+26 -21
View File
@@ -1,15 +1,35 @@
# Bulletproof Containers for Beta
# Bulletproof Containers
**Status**: plan agreed 2026-04-22, implementation started.
**Target**: zero-manual-intervention container lifecycle for the beta launch. A user installs, uninstalls, reboots, updates, or loses power — every combination must leave the node in a known-good state without SSH.
**Project memory**: `~/.claude/projects/-home-archipelago-Projects-archy/memory/project_reconcile_architecture.md`
**Failure log**: `~/.claude/projects/-home-archipelago-Projects-archy/memory/feedback_container_lifecycle_failure_modes.md`
**Status**: historical design record (agreed 2026-04-22). The *architecture* —
level-triggered, desired-state reconciliation — was adopted and is live. Several
specifics below were not built as written, so read this for the incident history
and the reasoning, not as a description of the code. For how the lifecycle
actually works today, read [Container lifecycle](container-lifecycle.md).
What became of the plan, verified against the tree:
| Item | Outcome |
|---|---|
| Level-triggered reconciler | ✅ Shipped, but as `container/boot_reconciler.rs` + `container/prod_orchestrator.rs`. The `core/archipelago/src/reconcile/` module laid out below (`desired.rs`/`current.rs`/`diff.rs`/`apply.rs`/…) **was never created** — no file in it exists |
| FM5 post-OTA probe + auto-rollback | ✅ Shipped — `update-pending-verify.json` (`update.rs:100`) |
| FM4 `host.archipelago` alias | ✅ Shipped — `AddHost=host.archipelago:10.89.0.1` in generated units |
| FM1/FM3 Quadlet ownership | ◐ Partial. Companion UIs run as Quadlet units; **main app containers do not** — `use_quadlet_backends` still defaults false, so the "v1.7.48+ full migration" below has not happened |
| FM2 bitcoin.conf drift | ◐ Solved differently. There is no `reconcile::derived::render_bitcoin_conf`; instead bitcoind is run with an explicit `-conf` derived from secrets at each start and stale datadir configs are removed (`remove_stale_bitcoin_conf`) |
| FM6 podman corrupt-state self-heal | ❌ **Not implemented.** No `podman system renumber` recovery, no startup probe for "invalid internal status". The failure that made a node unreachable in 2026-04 would still need manual SSH |
Note also that the unit paths below say `/etc/containers/systemd/`; units are
actually written per-user to `~/.config/containers/systemd/`
(`quadlet.rs:DEFAULT_REL_UNIT_DIR`), since the whole path is rootless.
**Target**: zero-manual-intervention container lifecycle. A user installs,
uninstalls, reboots, updates, or loses power — every combination must leave the
node in a known-good state without SSH.
---
## Why we're doing this
The v1.7.38 and v1.7.39 rollouts on 2026-04-22 exposed a cluster of container-lifecycle failures that required manual SSH recovery on every affected node (.116, .198, .228, .253). If a user had been on those nodes, they'd have been stuck with "can't reach" or 500 errors and no path forward. We can't ship beta with this class of failure on the table.
The v1.7.38 and v1.7.39 rollouts on 2026-04-22 exposed a cluster of container-lifecycle failures that required manual SSH recovery on every affected node. If a user had been on those nodes, they'd have been stuck with "can't reach" or 500 errors and no path forward. We can't ship beta with this class of failure on the table.
The pattern under every failure: **the canonical source of truth had the right answer, but derived state drifted away from it and nothing noticed or fixed it.**
@@ -297,18 +317,3 @@ Ordered by likelihood × severity:
### Tor
- [rend-spec-v3](https://github.com/torproject/torspec/blob/main/rend-spec-v3.txt) — descriptor lifetime + republish cadence
- [stem](https://stem.torproject.org/) — Python Tor controller for `HS_DESC UPLOADED` waits
---
## To resume
1. Read project memory: `~/.claude/projects/-home-archipelago-Projects-archy/memory/project_reconcile_architecture.md`
2. Read failure-mode memory: `~/.claude/projects/-home-archipelago-Projects-archy/memory/feedback_container_lifecycle_failure_modes.md`
3. Check task list for current release (should start with v1.7.41)
4. Current state on fleet as of 2026-04-22:
- All 4 mirrors (tx1138, gitea-local, .160, .168) synced to v1.7.40-alpha
- .116, .198, .228, .253 healed manually via `systemd-run chmod 755 /opt/archipelago/web-ui`
- .228 still has stale `bitcoin.conf` rpcauth (regenerated during triage; will drift again until v1.7.43)
- .228 UI companions (archy-bitcoin-ui, archy-lnd-ui) keep vanishing (Quadlet migration in v1.7.45+ fixes)
- .160 Gitea required `podman system renumber` recovery (v1.7.44 automates this)
5. Implementation is in progress on `main` branch — next edit is `core/archipelago/src/update.rs` for v1.7.41.
docs/combined-test-plan-2026-07-22.md
-173
View File
@@ -1,173 +0,0 @@
# Combined test session — 2026-07-22 batch (one sitting)
Staged on **framework-pt** (`100.65.115.109`) AND **archi thinkpad** (this
machine's node) so everything can be tested in one pass. Items marked ✅ were
already verified by the agent on a node; ❑ items need a human.
**What's in this batch:** mesh message/DM persistence across restarts ·
first-message + DM announce fix · 15s announce poll · radio hot-swap modal
(probe / keep-as-is / apply-settings) · whisper beam-1 (release-gated, see §D) ·
real-time wallet push (0-conf tx shows in seconds) · calm Lightning
"still starting" notice · external tx-explorer fallback with consent modal +
wallet-settings On-chain tab · apps open ABOVE modals with the launch
animation · mempool installs no longer blocked by a resyncing ElectrumX ·
[pending: other agents' two push sets — section F fills in when their code
lands].
## H. Wallet & explorer (new — test on the thinkpad node, it's pruned)
1. ❑ **Real-time tx display:** send a small on-chain amount to this node's
wallet → the balance and the yellow "unconfirmed" transaction appear
within a few seconds of broadcast, no refresh, no wallet action.
2. ❑ **External explorer consent:** with no local Mempool app running, tap a
transaction → amber consent modal explains it opens on another node's
mempool (default tx1138.com, placeholder mempool.guide, editable) →
Open Explorer opens `<explorer>/tx/<hash>` in a new tab. Tick "don't ask
again" and confirm the next tap opens directly.
3. ❑ **Wallet Settings → On-chain tab:** explorer URL editable, warning shown,
"don't warn" toggle; tabs now read Channels / Cashu / Fedi / Ark / On-chain
and fit on one row (check mobile too).
4. ❑ **Modal → app animation:** on a node WITH Mempool running, open
Transactions and tap a tx → the Mempool app animates in ABOVE the modal
(previously loaded invisibly underneath); closing it returns to the modal.
5. ❑ **Lightning "still starting":** right after a node restart, try opening a
channel → either it just works (silent retry) or a calm amber ⏳ notice
appears — never the red "Failed to connect to peer" error.
---
## A. Staged state (agent-verified before you start)
- ✅ Dev binary (persistence + announce seeder + hot-swap) on
`/usr/local/bin/archipelago`, service healthy, no crash-loop.
- ✅ Frontend bundle with the new device modal at `/opt/archipelago/web-ui`.
- ✅ Seeder re-ran: `automations.yaml` upgraded v1→v2 (first-message announce),
`configuration.yaml` rest block at `scan_interval: 15`, HA restarted clean.
- ✅ `mesh-messages.json` persisting + restored across a service restart.
- ✅ `mesh.probe-device` returns real firmware details for the plugged stick.
## B. Mesh history survives restarts (the "messages go missing" fix)
1. ❑ Open Mesh chat — your existing DM/channel history from today is visible.
2. ❑ Send one channel message and one DM (either direction).
3. ❑ Reboot the whole node (not just the service). After it's back: history
still there, including the two new messages, correct timestamps/senders.
4. ❑ Send a NEW message to another node right after the reboot and confirm the
other side receives it (this exercises the send-seq fix — before it, the
first post-reboot sends were silently dropped by peers as replays).
## C. Speaker announcements
1. ❑ Have another node send a **public channel** message → speaker announces
sender + text within ~15s (was ~30s).
2. ❑ Have another node send you a **DM** → speaker announces it the same way.
3. ❑ Restart Home Assistant (or the node) → the last old message is NOT
re-announced (no announce storm).
4. ❑ (First-message case — the original bug — only reproducible on a node with
an empty history: optional, covered by agent verification of the guard.)
## D. Voice (regression + speed)
1. ❑ "Hey Jarvis, what's the block height" and one fuzzy phrasing — same
correct answers as before (no behavior change is the pass condition).
2. ⓘ The ~45% faster speech-to-text (whisper beam-1) ships via the **signed
catalog in the release** — it is NOT on the node during this test session.
Benchmarked on this exact hardware: identical transcripts, 0.94s → 0.51s.
## E. Radio hot-swap modal (your Reticulum stick is already plugged in)
1. ❑ Open the web UI anywhere — within ~30s a "Mesh Radio Detected" modal
appears showing the stick on `/dev/ttyACM0`, with a card of what's on it
(firmware badge: Reticulum RNode / MeshCore / Meshtastic + current
name/region/channels where the firmware exposes them).
2. ❑ Press **Keep As Is** → mesh connects using the radio exactly as flashed
(check Mesh → Device tab: connected, firmware type correct; nothing on the
radio changed).
3. ❑ Unplug the stick, plug the old MeshCore one → the modal appears AGAIN
(every plug re-triggers, same or different /dev path).
4. ❑ This time press **Set Up with Archipelago Settings** → second screen
shows channel `archipelago`, your region, and the node's RF params (the
validated Portugal preset on this fleet) BEFORE anything is written;
confirm → radio provisions and joins the mesh.
5. ❑ Swap sticks once more with no UI interaction except "Keep As Is" — chat
still works end-to-end afterwards (hot-swap without ceremony).
## F. Companion pairing + mobile onboarding (other agent — push set #1, MERGED)
1. ❑ Companion app: pair with the node via the new named QR (device tokens) —
pairing completes instantly, device appears in the paired-devices list.
2. ❑ Remote access now rides the embedded FIPS mesh (WireGuard replaced):
with the phone OFF the node's WiFi, the companion still reaches the node.
3. ❑ The reworked mobile onboarding/intro overlay screens flow correctly on
first launch of the new APK (in-tarball APK is the 27MB build).
4. ❑ (Push set #2 from the other agents is still pending — the release waits
for it; this staged build does NOT include it yet.)
## G. Quick regressions
1. ❑ Pine launcher page (:10380) still shows the live node card; "Connect
Pine to WiFi" button loads without JS errors.
2. ❑ Mobile Home: wallet card directly under My Apps (G5 from the voice epic).
3. ❑ Mesh RF settings panel (Mesh → Device) still loads and saves.
## H. LoRa radio firmware flashing (Heltec V3/V4, new — extends Section E)
Full v1 scope is 3 firmware families × 2 boards (6 cells); mark each cell
tested on real hardware vs. code-reviewed only as this is run.
1. ❑ From the hot-swap modal's step 1 (device already probed), press
**Flash Firmware…** → new step shows firmware-family + board pickers and
the erase-confirmation checkbox; "Erase & Flash Now" stays disabled until
family, board, AND the checkbox are all set.
2. ❑ Confirm what's currently on the test stick via the existing probe
BEFORE flashing it — don't flash the only known-good device without a
fallback board on hand.
3. ❑ Prefer a spare Heltec V3/V4 for the first destructive erase+flash run;
only exercise a primary/in-use stick once the flow is proven safe.
4. ❑ MeshCore → Heltec V3: erase + write completes, progress bar and log
tail update live, ends at "Flash complete".
5. ❑ Meshtastic → Heltec V3: same, using the extracted `*.factory.bin` from
the esp32s3 release zip.
6. ❑ Reticulum/RNode → Heltec V3: `archy-rnodeconf --autoinstall` path
completes (no raw esptool erase/write step for this family — see
`mesh/flash.rs` doc comment).
7. ❑ Repeat 4-6 against a Heltec V4. Confirmed 2026-07-23 on real hardware:
V4 uses the ESP32-S3's native-USB JTAG/serial peripheral (vid:pid
303a:1001, generic to every native-USB ESP32-S3 board, not V4-specific)
— so unlike V3's CP2102 bridge chip, V4 is permanently NOT auto-matchable
by vid:pid. Board auto-detect should fail closed for it every time
(manual board selection required, "couldn't confirm automatically"
warning shown) — this is expected steady-state behavior, not a gap to
close later.
8. ❑ After a successful flash, the modal automatically re-probes and shows
the NEW firmware's badge/details — same as unplugging and replugging
(Section E item 3), but without physically touching the cable.
9. ❑ Deliberately test a failure path once (disconnect the board mid-write,
or point at a bad cached asset) — confirm the error surfaces in the
progress log AND that `docs/troubleshooting.md`'s "LoRa radio firmware
flash failed" recovery steps (BOOT+RST bootloader entry, manual esptool/
rnodeconf command) actually get the board back to a flashable state.
10. ❑ Cancel button only appears (and only works) while still in the
"Downloading firmware…" stage — once erasing/writing starts, no cancel
affordance is offered.
11. ❑ **Boot-loop regression (2026-07-23 incident)**: after a *failed* flash
(e.g. kill network access mid-download to force a failure), confirm the
mesh listener does NOT auto-resume — `journalctl -u archipelago` should
show a single `Leaving mesh listener stopped after failed flash` line
and then go quiet for that device, not a repeating `mesh::serial:
Opened serial port... Starting Meshcore handshake` cycle every few
seconds. Reconnect manually via the hot-swap modal afterward and confirm
it connects normally (the board itself should be untouched — the
download fails before esptool/rnodeconf ever runs).
12. ❑ Separately, force a device to flap connected/disconnected a few times
in under 20s each (e.g. a marginal USB connection) and confirm
`reconnect_delay` in the logs actually escalates (5s → 10s → 20s → ...)
rather than resetting to 5s on every attempt — see
`STABLE_SESSION_THRESHOLD` in `mesh/listener/mod.rs`.
---
After this passes: fold the batch + other agent's work into the next release
(OTA binary + frontend tarball + catalog regen/sign/publish for pine-whisper
3.4.2), then re-run `tests/lifecycle/run-gate.sh` on .228 (back online as
Tailscale `shorty-s`).
docs/companion-pairing-qr.md
+2 -2
View File
@@ -30,7 +30,7 @@ Query parameters:
| param | required | meaning |
|-------|----------|---------|
| `v` | yes | Payload version, currently `1`. Reject/ignore unknown majors gracefully — show "please update the app". |
| `url` | yes | Full origin the app should connect to, scheme included: `https://demo.archipelago-foundation.org`, `http://archipelago.local`, `http://192.168.1.228`, etc. No trailing slash guaranteed either way — normalize. |
| `url` | yes | Full origin the app should connect to, scheme included: `https://demo.archipelago-foundation.org`, `http://archipelago.local`, `http://192.0.2.10`, etc. No trailing slash guaranteed either way — normalize. |
| `name`| no | Display name for the server entry. Real nodes send the configured server name, or `My Archipelago` when it's still the factory default. |
| `tok` | no | **Device token** minted via `auth.createDeviceToken` when the QR is rendered. The app logs in with `{"method":"auth.login","params":{"token":"…"}}` — same endpoint, same rate limiter, skips TOTP (the token was minted from an authenticated session). Long-lived until re-minted (re-showing the pair screen replaces the `companion` token) or revoked (`auth.revokeDeviceToken`). Scan → instantly connected, no typing. |
| `pw` | no | Login password. **Only present in the public demo** (shared demo password `entertoexit`). Real nodes never embed a password — the frontend doesn't have it. |
@@ -43,7 +43,7 @@ Query parameters:
Examples the web UI actually emits:
- Demo: `archipelago://pair?v=1&url=https%3A%2F%2Fdemo.archipelago-foundation.org&pw=entertoexit`
- Real node, browsed via LAN IP: `archipelago://pair?v=1&url=http%3A%2F%2F192.168.1.228`
- Real node, browsed via LAN IP: `archipelago://pair?v=1&url=http%3A%2F%2F192.0.2.10`
- Real node kiosk (UI runs on localhost, so it advertises the mDNS name from
`system.get-hostname`): `archipelago://pair?v=1&url=http%3A%2F%2Farchipelago.local`
docs/container-architecture.html
-4279
View File
File diff suppressed because it is too large Load Diff
docs/container-lifecycle.md
+108
View File
@@ -0,0 +1,108 @@
# Container lifecycle
How Archipelago keeps apps in the state you asked for — install, start, stop,
restart, uninstall — and how it self-heals without ever resurrecting something
you deliberately stopped. Source of truth:
`core/archipelago/src/container/prod_orchestrator.rs` and
`core/archipelago/src/container/boot_reconciler.rs`.
## The model: level-triggered, not fire-and-forget
Archipelago does not start a container and hope. A long-running **reconciler**
compares *desired state* (what the manifests and your explicit choices say
should be running) against *actual state* (what podman reports) and repairs the
difference. It is **level-triggered**: it acts on the current gap every tick, not
on a one-time event, so a container that dies, a unit that vanishes, or a reboot
that clears everything are all just "the gap is non-zero, close it".
The reconciler is spawned once at boot (`BootReconciler`) after an initial
`adopt_existing()` pass, and runs every **30 seconds**. It finishes an in-flight
pull or build before honouring a shutdown signal — it is never interrupted
mid-operation.
Concurrency: each app has its own async mutex guarding all mutating operations
against the reconciler, so a manual `stop` and a reconcile tick can't race, but
reconciles across different apps still run without serialising against each
other.
## Desired state has three inputs
For each app the reconciler asks: *should this be running right now?* The answer
comes from three durable signals, checked in this order:
1. **Explicitly user-stopped** (`user-stopped.json`). If you stopped an app, its
id is recorded and the reconciler leaves it down — it is **not** a gap to
repair. Cleared when you start it again. This is what makes a stop *stick*
across restarts and reboots.
2. **Explicitly uninstalled** (`user-uninstalled.json`). Same idea for uninstall:
a baseline app you removed stays removed, so self-heal can't reinstall it.
3. **Otherwise, the manifest set** — every catalog/disk app that isn't stopped or
uninstalled should be running.
Dependencies are pulled in: an app that is up requires its declared
dependencies, so they are kept up too — but a dependency you explicitly stopped
still stays stopped.
## The operations
All go through the orchestrator, all take the per-app lock, all are idempotent:
| Operation | What it does |
|-------------|--------------|
| **adopt** | At boot, take ownership of a pre-existing container **by name** rather than recreating it — preserves data, ports and identity across a daemon restart. |
| **install** | Materialise secrets → ensure image (build from a local Dockerfile or use a pre-pulled image) → create and start the container (via Quadlet where enabled). |
| **start / stop** | Bring the container up/down and record the desired-state change. A stop writes the app to `user-stopped.json`. |
| **restart** | Stop then start, preserving the container's data and identity. |
| **remove** | Stop and remove the container, **preserving `/var/lib/archipelago/<app>`, secrets, credentials and ports** — a reinstall or upgrade lands on the same data. |
| **upgrade** | Recreate at a new image while preserving data (see the version rules below). |
| **health** | Report the container's health from its declared `health_check`. |
## Self-heal vs. respecting your choice
The one rule that ties it together: **self-heal must never override a deliberate
stop or uninstall.**
- A container that disappeared while its siblings run — a wedged teardown, a
reboot that cleared it — is a hole to repair, and the reconciler rebuilds it
from the durable "was running" snapshot.
- A container that is down because you stopped or uninstalled it is a *choice*,
and the reconciler leaves it alone.
A small set of **baseline apps** are expected to exist from first boot and
self-heal when their container is missing — but the `user_stopped` /
`user_uninstalled` gates are checked first, so even a baseline app you turned off
stays off. Getting this wrong in either direction is a real bug: resurrecting a
stopped app ignores the operator, and failing to rebuild a crashed one is the
fire-and-forget failure the whole design exists to remove.
## Migrations never destroy data
Any recreate path — upgrade, reinstall, repair — preserves the app's data
directory, its generated secrets, its credentials, its ports, and the container
name used for adoption. An update that would roll a version *backwards* is
refused (see the version guard in `container::image_versions`): the update button
never offers a lower version than what is running, so a stale record cannot turn
into a downgrade. Version pins are honoured — a pinned app is not "updated" out
from under the operator by the catalog.
## Inspecting lifecycle state
```bash
# what podman actually has — run as the archipelago service user (rootless)
podman ps -a --format '{{.Names}}\t{{.Status}}'
# the durable desired-state signals
cat /var/lib/archipelago/user-stopped.json
cat /var/lib/archipelago/user-uninstalled.json
# the reconciler's decisions. archipelago.service is a SYSTEM unit that runs
# as User=archipelago (WantedBy=multi-user.target), so this is not --user —
# unlike the companion Quadlet units, which are per-user.
sudo journalctl -u archipelago | grep -iE 'reconcile|adopt|install|user.stopped'
```
## Related
- [Manifest → Quadlet unit](quadlet-compilation.md) — how the unit the reconciler manages is generated
- [App secrets](secrets.md) — the `ensure_generated_secrets` tick that runs before start
- [App Manifest Specification](app-manifest-spec.md) — `health_check`, `dependencies`, `restart` fields
docs/demo-build-info.md
+2 -2
View File
@@ -3,7 +3,7 @@
**Status:** implemented & deployable (2026-07-14)
**Branch:** `main` — the demo machinery was merged from the old `demo-build`
branch and now lives on main, pushed to
`gitea-vps2` = `http://146.59.87.168:3000/lfg2025/archy.git`.
`gitea-vps2` = `https://source.archipelago-foundation.org/lfg2025/archy.git`.
A public, click-to-play demo of the Archipelago UI, 100% mock-data driven,
multi-visitor, deployed via Portainer. See also `docs/archive/demo-deployment-design.md`
@@ -17,7 +17,7 @@ Build-from-repo (works today, no registry needed):
| Field | Value |
|-------|-------|
| Repository URL | `http://146.59.87.168:3000/lfg2025/archy.git` |
| Repository URL | `https://source.archipelago-foundation.org/lfg2025/archy.git` |
| Reference | `refs/heads/main` |
| Compose path | `docker-compose.demo.yml` |
| Auth | user `lfg2025`, password = Gitea token |
docs/developer-guide.md
+10 -16
View File
@@ -54,11 +54,10 @@ archy/
│ ├── vite.config.ts
│ └── package.json
├── scripts/ # Deployment and utility scripts
│ ├── deploy-to-target.sh # Main deploy script
│ ├── first-boot-containers.sh # ISO first-boot setup
│ └── run-tests.sh # CI test runner
├── image-recipe/ # ISO build configuration
│ ├── build-auto-installer-iso.sh
│ ├── build-debian-iso.sh
│ └── configs/ # Nginx, systemd configs
├── docs/ # Documentation
│ ├── architecture.md
@@ -66,7 +65,7 @@ archy/
│ ├── marketplace-protocol.md
│ └── multi-node-architecture.md
├── apps/ # App manifests (YAML)
├── CLAUDE.md # AI development instructions
├── CLAUDE.md # Contributor guide (invariants, build/verify)
└── docs/ROADMAP.md # Project roadmap
```
@@ -91,19 +90,16 @@ The dev server at `http://localhost:8100` uses a mock backend.
### Deploying Changes
Release and host-integration builds should run on Linux. The deploy script rsyncs
source to a configured Linux target and builds there.
Release and host-integration builds should run on Linux. Build the backend and
frontend on the target, or cross-build and copy the artifacts across:
```bash
# Deploy to the configured primary target (builds backend + frontend, restarts services)
./scripts/deploy-to-target.sh --live
# Deploy to both configured targets
./scripts/deploy-to-target.sh --both
cd core && cargo build --release
cd neode-ui && npm ci && npm run build
```
The deploy script:
1. Rsyncs source to the server
A deploy then:
1. Copies the build output to the node
2. Builds Rust backend on the server (`cargo build --release`)
3. Builds Vue frontend (`npm run build`)
4. Copies artifacts to production paths
@@ -203,7 +199,6 @@ async myAction(params: { name: string }): Promise<{ ok: boolean; result: string
### 5. Deploy and Test
```bash
./scripts/deploy-to-target.sh --live
curl -X POST http://<node-host>/rpc/v1 \
-H "Content-Type: application/json" \
-b "archipelago_session=YOUR_SESSION" \
@@ -312,6 +307,5 @@ mod tests {
1. Create a feature branch: `git checkout -b feature/my-feature`
2. Make changes following the standards above
3. Test locally: `cd neode-ui && npm test`
4. Deploy to dev server: `./scripts/deploy-to-target.sh --live`
5. Verify on your configured development target
6. Commit with conventional format: `feat: add my feature`
4. Verify on an Archipelago node
5. Commit with conventional format: `feat: add my feature`
docs/dht-distribution-design.md
+11 -4
View File
@@ -1,6 +1,13 @@
# DHT / Peer-Distributed Content Design
**Status:** Design (no code yet) · **Date:** 2026-06-16 · **Author:** archipelago + Claude
**Status:** partially implemented — **not** "no code yet" as this line previously
read. `core/archipelago/src/swarm/` exists (`mod.rs`, `iroh_provider.rs`,
`paid.rs`, `paid_alpn.rs`, `payment.rs`) along with `content_hash.rs`, behind the
**default-off** `iroh-swarm` cargo feature (`Cargo.toml:21` — the iroh/iroh-blobs
deps are optional and only pulled in by that feature). `config.swarm_enabled`
gates it at runtime and also defaults off, so a stock build ships this inert.
Treat the phases below as design; check the feature flag before assuming a phase
is live. · **Date:** 2026-06-16 · **Author:** archipelago + Claude
## 1. Purpose
@@ -28,7 +35,7 @@ origin":
### OTA (`core/archipelago/src/update.rs`)
- Manifest at `DEFAULT_UPDATE_MANIFEST_URL` (`update.rs:67`) = vps2 OVH
(`146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json`).
(`source.archipelago-foundation.org/lfg2025/archy/raw/branch/main/releases/manifest.json`).
- `check_for_updates()` (`:565`) walks an operator mirror list (`default_mirrors()` `:105`,
`load_mirrors()` `:123`), origin-rewrites component URLs to the chosen mirror
(`rewrite_manifest_origins()` `:227`).
@@ -65,9 +72,9 @@ origin":
### IndeeHub (the streaming target)
- Original platform (not a fork). Working source: `~/Projects/Indeedhub Prototype/`
(Vue 3 + NestJS). Submodule `146.59.87.168:3000/lfg2025/indeehub.git` (repointed off the retired host —
(Vue 3 + NestJS). Submodule `source.archipelago-foundation.org/lfg2025/indeehub.git` (repointed off the retired host —
needs a live remote). In `archy`: image-only, `apps/indeedhub/manifest.yml` pulls
`146.59.87.168:3000/lfg2025/indeedhub:1.0.0` (+ `-api`, `-ffmpeg`, postgres, redis,
`source.archipelago-foundation.org/lfg2025/indeedhub:1.0.0` (+ `-api`, `-ffmpeg`, postgres, redis,
minio, nostr-rs-relay).
- Streaming today: FFmpeg → **HLS (.m3u8 + AES-128 .ts segments)** in **MinIO**
(`indeedhub-private`/`-public`), metadata in Postgres, transcode queue in Redis,
docs/dual-ecash-design.md
+1 -1
View File
@@ -94,7 +94,7 @@ in sync with the manifest env. clientd is a **client, not the guardian** — it
under `image-recipe/_archived/` (likely stale); `first-boot-containers.sh`/`image-versions.sh`
are current.
- Image: build from source (no official image; `flake.nix` only) → push to vps2
`146.59.87.168:3000/lfg2025/fedimint-clientd:v0.4.0`.
`source.archipelago-foundation.org/lfg2025/fedimint-clientd:v0.4.0`.
### 5. Unified balance
`HomeWalletCard` ecash row = Cashu `wallet.ecash-balance` + Fedimint `wallet.fedimint-balance`.
docs/hotfix-process.md
-52
View File
@@ -1,52 +0,0 @@
# Hotfix Process
For critical bugs discovered after a tagged release.
## Severity Classification
| Level | Response Time | Examples |
|-------|--------------|---------|
| P0 — Critical | < 4 hours | Data loss, security vulnerability, node bricked |
| P1 — High | < 24 hours | App won't start, auth broken, major UI failure |
| P2 — Medium | < 72 hours | Non-critical feature broken, performance regression |
| P3 — Low | Next release | Cosmetic, minor UX, edge cases |
## Hotfix Workflow
### 1. Triage
- Reproduce the issue on dev server (192.168.1.228)
- Classify severity (P0-P3)
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
### 2. Fix
- Create branch: `hotfix/vX.Y.Z-description`
- Fix the issue with minimal code changes
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
- Verify fix on live server
### 3. Release
- Merge hotfix branch to `main`
- Tag: `vX.Y.Z` (increment patch version)
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
`releases/manifest.json` and signs it)
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
### 4. Communicate
- Update RELEASE-NOTES with hotfix details
- Note in CHANGELOG.md
## Monitoring Dashboards
- **Uptime monitor**: `/var/lib/archipelago/uptime-monitor/summary.json`
- **Soak test**: `/tmp/stability-test-*.log` on dev server
- **Health endpoint**: `http://192.168.1.228/health`
## Rollback
If a hotfix causes regressions:
1. The updater self-verifies after applying (health check on restart) and rolls the
binary back automatically if the new one fails to come up
2. Point `releases/manifest.json` back at the last-known-good version and push
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
docs/manifest-hooks-design.md
+1 -1
View File
@@ -3,7 +3,7 @@
**Status:** implemented through Phase 4 (see §6; updated 2026-07-08) — only declarative `pre_start` remains · originally Task #20
(indeedhub, netbird) off legacy Rust installers.
See `docs/PRODUCTION-MASTER-PLAN.md`, `docs/APP-PACKAGING-MIGRATION-PLAN.md`
See `docs/APP-PACKAGING-MIGRATION-PLAN.md`
("controlled hooks").
---
docs/marketplace-protocol.md
+169 -47
View File
@@ -10,6 +10,14 @@ purchases). What remains is maturation: publishing tooling and trust UX
own flatter format, **not** the runtime `apps/*/manifest.yml` schema
(`app-manifest-spec.md`).
> **The DID signature layer is implemented** as of 2026-08-08. `publish` signs
> with the node's Ed25519 identity key, `discover` verifies every manifest
> before caching it, and a manifest whose signature is *present but wrong* is
> dropped rather than listed at a lower score. See
> [Signing Protocol](#signing-protocol) for the exact preimage rules — they are
> normative, and an implementation that canonicalises differently will produce
> signatures this node rejects.
## Overview
Archipelago's community marketplace enables developers to publish app manifests to Nostr relays, where nodes discover and install them without a central app store. Trust is established through DID-signed manifests and community reputation.
@@ -37,7 +45,12 @@ Developer Node Nostr Relays User Node
## Manifest Schema
App manifests published to Nostr relays follow the existing `apps/{app-id}/manifest.yml` schema (see `docs/app-manifest-spec.md`), serialized as JSON within a Nostr event.
App manifests published to Nostr relays use the marketplace's own flatter JSON
schema — the `AppManifest` type in `marketplace.rs`, shown below — serialized
into the Nostr event's `content`. It is **not** the runtime
`apps/{app-id}/manifest.yml` schema in
[`app-manifest-spec.md`](app-manifest-spec.md); the two are separate types that
happen to share a name.
### Marketplace Manifest Fields
@@ -98,7 +111,7 @@ App manifests published to Nostr relays follow the existing `apps/{app-id}/manif
|-------|---------|-------------|
| `container.readonly_root` | true | Container root filesystem is read-only |
| `container.no_new_privileges` | true | Prevent privilege escalation |
| `container.run_as_user` | 1000 | UID to run as (must be > 1000) |
| `container.run_as_user` | 1000 | UID to run as (must be ≥ 1000) |
| `container.capabilities` | [] | Required Linux capabilities (drop all, add only needed) |
## Nostr Event Format
@@ -141,13 +154,21 @@ App manifests use **NIP-78 application-specific data** with event kind **30078**
### Publishing a Manifest
1. Developer creates/updates their app manifest
2. Serialize manifest as JSON
3. Compute SHA-256 hash of the serialized manifest
4. Sign the hash with the developer's DID key
5. Embed manifest + signature in Nostr event content
6. Sign the Nostr event with the node's secp256k1 key
2. `author.did` is filled in with the node's own `did:key` if empty. If it is
set to a **different** DID, publishing is refused — the node can only sign as
itself, and broadcasting a manifest every verifier will reject helps nobody
3. Canonicalise the manifest without `signatures` and SHA-256 it (see
[Signing Protocol](#signing-protocol))
4. Sign the digest with the node's Ed25519 identity key and attach `signatures`
5. Embed the signed manifest as the Nostr event content
6. Sign the Nostr event with the node's secp256k1 Nostr key
7. Publish to all configured Nostr relays
Note the two distinct keys: the **Ed25519 identity key** proves *authorship of
the manifest* and is what `author.did` names; the **secp256k1 Nostr key** proves
*who sent this event*. They are separate on purpose — relaying is not
authorship, and only the first survives being copied between relays.
### Discovering Manifests
1. Node queries configured relays with filter:
@@ -172,13 +193,29 @@ App manifests use **NIP-78 application-specific data** with event kind **30078**
Each discovered app receives a trust score (0-100) based on:
| Factor | Weight | Description |
|--------|--------|-------------|
| **DID Verification** | 30 | Manifest is signed by a valid DID key |
| **Relay Consensus** | 20 | Manifest found on multiple independent relays |
| **Federation Trust** | 20 | Developer's DID is in the user's federation network |
| **Version History** | 15 | App has multiple published versions (shows maintenance) |
| **Security Compliance** | 15 | Manifest follows all security requirements |
This table is `calculate_trust_score()` in `marketplace.rs`. What each factor
actually checks:
| Factor | Max | What is checked |
|--------|-----|-----------------|
| **Identity proven** | 30 | The manifest carries a `valid` DID signature — the author demonstrated control of the key `author.did` encodes. Requires key material; cannot be faked by choosing a string |
| **Relay consensus** | 20 | Graduated, and never zero: 1 relay → 5, 2–3 → 12, 4+ → 20 |
| **Federation trust** | 20 | `author.did` is in the user's federated DID list **and** identity is proven. Both halves are required — see below |
| **Provenance** | 15 | 10 for a 3-part semver `version`, 5 for a non-empty `repo_url`. Nothing counts published versions |
| **Security compliance** | 15 | 15 when `validate_manifest()` returns no issues, 5 when it returns 1–2, 0 otherwise |
Both identity-derived factors hang off the signature, which is the point:
- Before, "DID present" was `did.starts_with("did:")`, so an unsigned manifest
with a plausible-looking DID string and a pinned image scored 65 — *Community*
tier — on no cryptography whatsoever. It now scores 35, *Unverified*.
- Federation trust is gated too. An unverified `author.did` is just a string the
publisher chose, so an attacker could otherwise copy the DID of a peer the
user federates with and collect 20 points for impersonating precisely the
party the user trusts most.
An unsigned publisher is not punished beyond losing those points: `missing` is a
normal state, and such apps still appear.
### Trust Tiers
@@ -212,19 +249,44 @@ When a developer's DID appears in the user's federation network (trusted peer),
### Manifest Signing (DID Layer)
**Normative.** These rules define the signed preimage byte-for-byte. An
implementation that canonicalises differently will produce signatures this node
rejects, so they are worth following exactly.
```
1. Serialize manifest to canonical JSON (sorted keys, no whitespace)
2. Compute: manifest_hash = SHA-256(canonical_json)
3. Sign: did_signature = Ed25519_Sign(did_private_key, manifest_hash)
4. Attach to manifest:
1. Take the manifest with `signatures` REMOVED (a signature cannot cover the
field that holds it; omit the key entirely rather than setting it null).
2. Canonicalise to JSON:
- every object's keys sorted lexicographically, recursively;
- no insignificant whitespace;
- arrays keep their order.
3. manifest_hash = SHA-256(canonical_json_bytes)
4. did_signature = Ed25519_Sign(author_private_key, manifest_hash)
^ the signature covers the 32 RAW DIGEST BYTES, not the "sha256:..."
string and not the JSON itself.
5. Attach:
{
"signatures": {
"manifest_hash": "sha256:<hex>",
"did_signature": "<base64>"
"manifest_hash": "sha256:<64 lowercase hex chars>",
"did_signature": "<standard base64, RFC 4648 §4, with padding>"
}
}
```
The signing key MUST be the Ed25519 key that `author.did` encodes — `author.did`
is a `did:key` whose multibase body is `0xed01 || <32-byte public key>`. A
publisher signing with any other key produces a manifest that verifies as
`invalid` and is dropped.
**Why canonicalisation is required and not cosmetic.** `container.env` is a map,
and map iteration order is not stable across processes or implementations. Sign
the serialiser's natural output and the same manifest hashes differently between
runs, so signatures fail at random rather than never — much harder to diagnose
than a clean rejection. Sorting keys removes the ambiguity.
`archipelago` implements this in `marketplace::canonical_signing_bytes` /
`sign_manifest` / `verify_manifest_signature`.
### Event Signing (Nostr Layer)
Standard NIP-01 Schnorr signature over the event ID (hash of serialized event fields). This is handled by the Nostr client library.
@@ -233,16 +295,32 @@ Standard NIP-01 Schnorr signature over the event ID (hash of serialized event fi
```
Receiving Node:
1. Verify Nostr event signature (NIP-01) → Proves event authenticity
2. Extract manifest JSON from event content
3. Compute SHA-256 of manifest content
4. Compare with manifest.signatures.manifest_hash → Proves content integrity
5. Resolve DID document for manifest.author.did
6. Verify did_signature with DID public key → Proves developer identity
7. Check container.image tag is pinned (not :latest)
8. Validate security fields meet minimums
1. Verify Nostr event signature (NIP-01) → event authenticity [IMPLEMENTED]
2. Extract manifest JSON from event content [IMPLEMENTED]
3. Canonicalise the manifest without `signatures`, SHA-256 it [IMPLEMENTED]
4. Compare with manifest.signatures.manifest_hash → content integrity [IMPLEMENTED]
5. Resolve author.did (did:key) to its Ed25519 public key [IMPLEMENTED]
6. Verify did_signature over the digest → author identity [IMPLEMENTED]
7. Check container.image tag is pinned (not :latest) [ADVISORY]
8. Validate security fields meet minimums [ADVISORY]
```
Steps 3–6 are `verify_manifest_signature()`, which returns one of three verdicts
rather than a boolean:
| Verdict | Meaning | What discovery does |
|---|---|---|
| `valid` | Hash matches the content **and** the key named by `author.did` signed it | Listed; earns the identity-derived trust points |
| `missing` | No `signatures` block | Listed, but scores **zero** on identity and federation. An unsigned publisher is unproven, not hostile |
| `invalid` | A `signatures` block is present and wrong — tampered, corrupt, or signed by another key | **Dropped entirely**, with the reason logged. Never cached, never installable |
That `invalid` handling is deliberate: a broken signature is not a low-quality
manifest, it is a forged or corrupted one, so it fails closed rather than
appearing with a scary badge someone can click past.
Steps 7–8 still run, but `validate_manifest()` returns a list of *issues* that
feed the trust score — they do not block discovery or installation.
## RPC Endpoints
### Marketplace Discovery
@@ -250,30 +328,67 @@ Receiving Node:
| Method | Description | Auth |
|--------|-------------|------|
| `marketplace.discover` | Query relays for app manifests, verify, score, return sorted | Local |
| `marketplace.publish` | Publish an app manifest to configured relays | Local |
| `marketplace.publish` | Sign the manifest with this node's identity key, then publish to configured relays | Local |
| `marketplace.get-manifest` | Get full manifest for a specific app by ID | Local |
| `marketplace.verify` | Verify a manifest's signatures and security compliance | Local |
| `marketplace.verify` | Check a manifest's DID signature and security compliance without publishing it | Local |
`marketplace.verify` returns the signature verdict separately from the advisory
policy issues, because they mean different things:
```json
{
"signature": { "status": "invalid", "reason": "did_signature does not verify against author.did" },
"signature_valid": false,
"valid": true, // ← policy compliance only; NOT authenticity
"issues": [],
"trust_score": 35,
"trust_tier": "unverified"
}
```
`valid` has always meant "passes the advisory security checks". Read
`signature_valid` for authenticity. Discovered apps carry the same verdict in
their `signature` field.
### Manifest Management
| Method | Description | Auth |
|--------|-------------|------|
| `marketplace.list-published` | List manifests published by this node | Local |
| `marketplace.unpublish` | Remove a published manifest from relays | Local |
### Purchases
| Method | Description | Auth |
|--------|-------------|------|
| `marketplace.create-invoice` | Create a Lightning BOLT11 invoice for a paid app | Local |
| `marketplace.check-payment` | Poll whether an invoice has settled | Local |
`marketplace.unpublish` was specified here but **never implemented** — the
string appears nowhere in the codebase, and there is no dispatcher entry. NIP-33
replaceable events mean an unpublish would have to be a tombstone/replacement
rather than a delete, which is presumably why it stalled.
## Security Requirements
### Container Security Enforcement
Before installing a community app, the node validates:
`validate_manifest()` checks the following and returns them as a list of issues.
**These are score inputs, not gates** — a manifest that fails all of them is
still discoverable and installable, it just scores 0 on the security factor:
1. **No `latest` tag**: Image must use a specific version tag
2. **Read-only root**: `readonly_root` must be true (or explicitly overridden by user)
3. **No root**: `run_as_user` must be > 1000
4. **No new privileges**: `no_new_privileges` must be true
5. **Minimal capabilities**: Only allowed capabilities are accepted (CHOWN, NET_BIND_SERVICE, etc.)
6. **No host networking**: Apps cannot use `--network host`
7. **Volume restrictions**: Apps cannot mount system paths (/, /etc, /var, /usr)
2. **Read-only root**: `readonly_root` should be true
3. **No root**: `run_as_user` must be **≥ 1000** (the code's bound; the example
manifest above uses exactly `1000`)
4. **No new privileges**: `no_new_privileges` should be true
Items previously listed here — a capability allow-list, a host-networking ban,
and system-path mount restrictions — are **not** part of marketplace validation.
Those rules exist, but they live in the runtime manifest parser
(`core/container/src/manifest.rs`, see [`app-manifest-spec.md`](app-manifest-spec.md))
and apply to `apps/*/manifest.yml`, which is a different schema from the
marketplace manifest. Closing that gap is part of the pre-third-party-publishing
work.
### Image Verification
@@ -316,22 +431,29 @@ Accessible from Settings or a "Developer" section:
```
/var/lib/archipelago/marketplace/
├── cache/
│ ├── manifests.json # Cached discovered manifests
│ └── trust-scores.json # Cached trust scores
├── published/
│ └── <app-id>.json # Manifests published by this node
└── config.json # Marketplace preferences (auto-refresh interval, etc.)
│ └── manifests.json # Cached discovered manifests, trust scores included
└── published/
└── <app-id>.json # Manifests published by this node
```
The earlier version of this tree also listed `cache/trust-scores.json` and
`config.json`. Neither is written: scores live on the cached entries themselves
(`MarketplaceCache`), and there is no marketplace preferences file.
## Implementation Notes
### Relay Query Strategy
1. Query all enabled relays in parallel (from `nostr_relays.rs` config)
1. Query all enabled relays in parallel (from `nostr_relays.rs` config), with a
10s connect timeout and a 20s fetch timeout per relay
2. Deduplicate manifests by `app_id` + `version`
3. If same manifest found on multiple relays, boost trust score
4. Cache results with 15-minute TTL
5. Background refresh every 30 minutes
3. If the same manifest is found on multiple relays, boost trust score
4. Write results to `cache/manifests.json`
Items 4–5 of the original design — a 15-minute cache TTL and a 30-minute
background refresh — are **not implemented**. The cache has no expiry and
nothing refreshes it on a timer; it is rewritten whenever
`marketplace.discover` runs.
### Version Comparison
docs/multinode-testing-plan.md
-69
View File
@@ -1,69 +0,0 @@
# Multinode / Fleet Testing Plan (separate from the single-node gate)
> **Scope split (2026-06-22):** the production test gate (`docs/PRODUCTION-MASTER-PLAN.md` §5,
> `tests/lifecycle/TESTING.md`) is now a **single-node criterion on .228**. Verifying the same
> lifecycle matrix across the rest of the fleet (.198 and the other testers) lives HERE and is run
> **after** the .228 single-node gate is green. This is intentionally NOT a blocker on the .228 gate.
## Why split it out
The lifecycle gate must be **run ON the node under test** — its bitcoin/companion/orphan/endpoint
checks use local `podman`/`systemctl`/`bitcoin-cli`/`curl`, not RPC to a remote host. Running it from
one host against another silently tests the *runner*. So "multinode" isn't "point the harness at N
hosts" — it's "run the on-node gate on each host," plus the genuinely cross-node concerns (federation,
mesh, transport, sync) that a single node can't exercise.
## How to run the gate on another node
Bats + jq usually aren't installed on ISO nodes. Bootstrap (one-time per node):
```
# from a host that has them (e.g. .116):
dpkg -L bats | grep -E '^/usr/(bin|lib|libexec)' | tar czf /tmp/bats.tgz -P -T - $(which jq)
tar czf /tmp/tests.tgz -C <repo> tests/lifecycle
scp /tmp/bats.tgz /tmp/tests.tgz <node>:/tmp/
# on the node:
sudo tar xzf /tmp/bats.tgz -P -C / # bats (jq here is dynamically linked — may need libs)
sudo curl -fsSL -o /usr/local/bin/jq \
https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 && sudo chmod +x /usr/local/bin/jq
mkdir -p /tmp/lifecycle-run && tar xzf /tmp/tests.tgz -C /tmp/lifecycle-run
cd /tmp/lifecycle-run/tests/lifecycle
ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=https ARCHY_PASSWORD=<node pw> \
ARCHY_ALLOW_DESTRUCTIVE=1 ARCHY_ITERATIONS=5 nohup ./run-gate.sh > /tmp/gate.log 2>&1 &
```
## Per-node preconditions (learned on .228)
- **Bitcoin must be fully synced + archival** (`initialblockdownload:false`, `pruned:false`).
test 83 reads the *real* `getblockchaininfo`, not the UI's headers-height. A node mid-IBD will
cascade-fail electrumx/lnd/btcpay/mempool even though the apps run.
- **Backends should be proper installs** (in `manifest_ids`), not adopted plain-podman left over
from ad-hoc `package.start`/cascade churn — otherwise companion self-heal and quadlet checks skew.
- **No stale per-app nginx proxy targets.** e.g. `/app/lnd/` must point at the lnd-ui port (18083),
not a stale `8081`. Repo code is correct; old node configs may be stale — re-check + regenerate.
- **No orphan quadlet units** (e.g. a `home-assistant.container` whose ContainerName ≠ the real
`homeassistant` container) — these wedge `systemctl --user` "activating" and fail the quadlet checks.
## Node roster (carry-over)
| Node | Role | Notes |
|------|------|-------|
| .228 | **single-node gate** (primary) | 14-app resilience node; bitcoin synced archival; gate GREEN. |
| .198 | fleet verify | was weak/loaded (load ~3–5) + **bitcoin mid-IBD** at split time → must finish syncing first; sshd wedges under concurrent SSH (use ONE session; gate uses HTTPS RPC so fine). |
| .5 / .120 | x250 testers (Tailscale) | flaky cellular; SSH via `tailscale nc` ProxyCommand. |
| .116 | dev/validation | local repo; its own bitcoin may be mid-IBD — do NOT treat as a gate target unless synced. |
## Cross-node concerns (only a multinode setup can test)
- Federation sync (Tor/FIPS transports), DID/contact federation, peer file fetch.
- Mesh (Meshtastic/MeshCore) + mesh-AI gating.
- Dual-ecash federation validation + networking-sats routing.
- DHT / iroh swarm distribution (origin-always-wins) once that dep lands.
## Sequence
1. Get the **.228 single-node gate green 5×** (master plan §5/§6) — DONE/in progress.
2. THEN: bring each fleet node to the preconditions above; run the on-node gate 5× per node.
3. THEN: the cross-node suites (federation/mesh/transport), tracked here.
This plan does not gate the v1.7.x single-node criterion; it is the next layer.
docs/operations-runbook.md
-366
View File
@@ -1,366 +0,0 @@
# Archipelago Operations Runbook
Quick reference for common operational tasks on Archipelago nodes.
**Primary node**: `192.168.1.228` (Arch 1)
**Secondary node**: `192.168.1.198` (Arch 2)
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
**Sudo**: use the node's sudo password (kept out of this doc — never commit credentials)
---
## 1. Check Node Health
```bash
# Quick health check (from any machine)
curl http://192.168.1.228/health # Should return "OK"
curl http://192.168.1.198/health
# Detailed system stats via RPC
curl -s -X POST -H "Content-Type: application/json" \
-d '{"method":"system.stats"}' \
http://192.168.1.228:5678/rpc/v1
# Check services
ssh archipelago@192.168.1.228
sudo systemctl status archipelago # Backend service
sudo systemctl status nginx # Web server
sudo systemctl status tor # Tor hidden services
```
## 2. Check Container Status
```bash
# List all containers
podman ps -a
# Running count
podman ps --format '{{.Names}}' | wc -l
# Find exited/crashed containers
podman ps -a --filter status=exited
# Container logs
podman logs {container-name} --tail 50
# Container resource usage
podman stats --no-stream
```
## 3. Fix Crashed Containers
```bash
# Restart a specific container
podman restart {container-name}
# If container won't start, check logs first
podman logs {container-name} --tail 100
# Remove and recreate (last resort)
podman rm -f {container-name}
# Then redeploy with: ./scripts/deploy-to-target.sh --live
# The health monitor auto-restarts containers every 60s
# Check its status:
sudo journalctl -u archipelago --grep="health_monitor" --no-pager -n 20
```
## 4. Add/Remove Federation Peers
```bash
# Generate invite code (on inviting node)
# Via UI: Federation page > Generate Invite
# Via RPC:
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.invite"}' \
http://localhost:5678/rpc/v1
# Join federation (on joining node)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.join","params":{"invite_code":"{code}"}}' \
http://localhost:5678/rpc/v1
# List peers
curl -s -X POST -H "Content-Type: application/json" \
-d '{"method":"federation.list-nodes"}' \
http://localhost:5678/rpc/v1
# Remove a peer
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"federation.remove-node","params":{"did":"{peer-did}"}}' \
http://localhost:5678/rpc/v1
```
## 5. Rotate Tor Address
```bash
# Delete current hidden service keys
sudo rm -rf /var/lib/tor/hidden_service/
sudo systemctl restart tor
# Wait for new hostname
sleep 15
sudo cat /var/lib/tor/hidden_service/hostname
# The backend picks up the new address automatically (30s refresh)
# Federation peers need to re-discover via sync
```
## 6. Create/Restore Backups
```bash
# Create encrypted backup (via RPC)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.create","params":{"passphrase":"your-passphrase","description":"manual backup"}}' \
http://localhost:5678/rpc/v1
# List backups
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.list"}' \
http://localhost:5678/rpc/v1
# Verify backup integrity
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.verify","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
http://localhost:5678/rpc/v1
# Restore (warning: overwrites current identity/data)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"backup.restore","params":{"id":"{backup-id}","passphrase":"your-passphrase"}}' \
http://localhost:5678/rpc/v1
# Backup files stored at: /var/lib/archipelago/backups/
```
## 7. Update the Node
```bash
# From development machine:
./scripts/deploy-to-target.sh --live # Deploy to .228
./scripts/deploy-to-target.sh --both # Deploy to both nodes
./scripts/deploy-to-target.sh --dry-run --live # Preview changes
# The deploy script:
# 1. Syncs code to target
# 2. Builds frontend (vue-tsc + vite)
# 3. Builds backend (cargo build --release)
# 4. Deploys binary, frontend, configs
# 5. Restarts services
# 6. Verifies health
```
## 8. Diagnose High CPU
```bash
# Check system load
uptime
# Find CPU-heavy processes
top -b -n 1 | head -15
# Check container CPU usage
podman stats --no-stream --format '{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
# Common causes:
# - Bitcoin IBD (initial block download): normal, takes days
# - Container crash loops: check `podman ps -a --filter status=exited`
# - mempool-electrs indexing: normal after Bitcoin sync
```
## 9. Diagnose High Memory
```bash
# Check memory
free -h
# Check swap usage
swapon --show
# Per-container memory
podman stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
# Check for OOM kills
dmesg --level=err,crit | grep -i oom
# Add swap if missing
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
```
## 10. Diagnose Disk Space
```bash
# Disk usage overview
df -h /
# Find large directories
sudo du -h --max-depth=2 /var/lib/archipelago/ | sort -rh | head -20
# Container image sizes
podman images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}'
# Clean unused images
podman image prune -a
# Clean old journal logs
sudo journalctl --vacuum-size=500M
```
## 11. Check Tor Connectivity
```bash
# Tor service status
sudo systemctl status tor
# Get onion address
sudo cat /var/lib/tor/hidden_service/hostname
# Test self-connection via Tor
curl --socks5-hostname 127.0.0.1:9050 http://$(sudo cat /var/lib/tor/hidden_service/hostname)/health
# Test cross-node Tor
curl --socks5-hostname 127.0.0.1:9050 http://{peer-onion}/health
```
## 12. Check DWN Sync
```bash
# DWN status (via RPC, needs auth)
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"dwn.status"}' \
http://localhost:5678/rpc/v1
# Trigger manual sync
curl -s -X POST -H "Content-Type: application/json" \
-H "Cookie: session={session}; csrf_token={csrf}" \
-H "X-CSRF-Token: {csrf}" \
-d '{"method":"dwn.sync"}' \
http://localhost:5678/rpc/v1
# Check message count
ls /var/lib/archipelago/dwn/messages/ | wc -l
```
## 13. Restart Services
```bash
# Restart backend only
sudo systemctl restart archipelago
# Restart nginx
sudo systemctl restart nginx
# Restart Tor
sudo systemctl restart tor
# Full service restart (backend + nginx)
sudo systemctl restart archipelago nginx
# Reboot (containers auto-recover via restart policy + health monitor)
sudo reboot
```
## 14. View Logs
```bash
# Backend logs
sudo journalctl -u archipelago --no-pager -n 100
# Follow logs in real time
sudo journalctl -u archipelago -f
# Nginx access log
sudo tail -f /var/log/nginx/access.log
# Nginx error log
sudo tail -f /var/log/nginx/error.log
# Container logs
podman logs {container-name} --tail 50 -f
```
## 15. Network Diagnostics
```bash
# Check listening ports
sudo ss -tlnp
# Check firewall rules
sudo ufw status verbose
# Required ports:
# 22 - SSH
# 80 - HTTP (nginx)
# 443 - HTTPS (nginx)
# 5678 - Backend API (localhost only, proxied by nginx)
# 8332 - Bitcoin RPC (container network only)
# 9050 - Tor SOCKS proxy (localhost only)
# If ports are blocked after reboot, re-add UFW rules:
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow from 10.88.0.0/16 # Podman container subnet
sudo ufw allow from 10.89.0.0/16 # Podman container subnet
```
## 16. Emergency: Node Won't Boot
If a node responds to ping but SSH/HTTP are down:
1. **Check UFW**: After reboot, UFW may block all ports
```bash
# If you have console access:
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw reload
```
2. **Check services**: SSH or nginx may not have started
```bash
sudo systemctl start ssh
sudo systemctl start nginx
sudo systemctl start archipelago
```
3. **Check disk**: If root filesystem is full, services won't start
```bash
df -h /
sudo journalctl --vacuum-size=200M
podman image prune -a
```
## 17. Run Tests
```bash
# Production lifecycle gate — run ON the node (uses local podman/systemctl):
tests/lifecycle/run-gate.sh # see tests/lifecycle/TESTING.md
ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh
# Cross-node suites (federation/mesh):
tests/multinode/smoke.sh # see docs/multinode-testing-plan.md
# E2E / post-install:
./scripts/run-e2e-tests.sh
./scripts/run-post-install-tests.sh
```
docs/phase4-streaming-ecash-plan.md
+10 -1
View File
@@ -1,6 +1,15 @@
# Phase 4+ — Paid swarm streaming & the IndeeHub "Archipelago" source
**Status:** PLAN / design (2026-06-17) · **Branch:** `agent-trust-wip` · not implemented
**Status:** PLAN / design (2026-06-17) · **Branch:** `agent-trust-wip` ·
**partly implemented — "not implemented" was stale.** The paid-serving half
landed on main: `core/archipelago/src/swarm/paid.rs` says in its own header that
it is "DHT distribution plan, Phase 4 step F", with `paid_alpn.rs` and
`payment.rs` alongside it, a `streaming::` module, and the
`streaming.list-services` / `configure-service` / `toggle-service` / `pay` /
`prepare-payment` RPCs. It is doubly default-off: the swarm needs the
`iroh-swarm` cargo feature plus `config.swarm_enabled`, and serving stays free
for everyone until the operator enables the `content-download` service. Check
those gates before assuming any step below is live or dead.
**Builds on:** `docs/dht-distribution-design.md` (Phases 0–3, swarm + Blossom), the
Phase 3 swarm work just landed (`swarm/`, `content_hash.rs`, `trust/`).
docs/pine-voice-release-test-plan.md
-60
View File
@@ -1,60 +0,0 @@
# Framework PT test plan — Pine voice epic (pre-release gate)
Target node: **framework-pt** (`100.65.115.109`, LAN 192.168.1.249). Run after
BOTH agents' work is merged, with the dev binary sideloaded and the signed
catalog (pine 1.3.0 + pine-openwakeword) published. Every ❑ must pass before
the release ritual starts. Items marked **(user)** need a human in the room.
## A. Deploy / prerequisites
- ❑ A1 Dev binary sideloaded, `archipelago` service active, no crash-loop in journal.
- ❑ A2 nginx self-heal added `location /api/pine/status` to every server block; `nginx -t` passes; nginx reloaded.
- ❑ A3 Signed catalog with pine 1.3.0 + pine-openwakeword live at the raw URL; node refreshed it (hourly sweep or "Check for updates").
## B. `/api/pine/status` endpoint
- ❑ B1 Public tier through nginx (`curl http://127.0.0.1/api/pine/status`): version, uptime, bitcoin height/sync_percent/peers, mesh peers. `lightning` null, `mesh_message` absent.
- ❑ B2 Wrong bearer token → still public-only (no balances). Correct token (from `/var/lib/archipelago/secrets/pine-status-token`) → lightning balances + latest mesh message present.
- ❑ B3 Reachable from inside the HA container via `host.containers.internal:80`.
- ❑ B4 Token file is 0600, owned by the service user.
## C. Stack / openwakeword container
- ❑ C1 Reconcile installs `pine-openwakeword` (wyoming-openwakeword 2.1.0), healthy on :10400.
- ❑ C2 Existing pine-whisper / pine-piper / pine were ADOPTED, not recreated — model data dirs untouched.
- ❑ C3 `archipelago` service restart → all four pine containers come back (crash-recovery stack spec).
- ❑ C4 UI: openwakeword listed under Services (no extra store card); Pine card shows 1.3.0.
## D. Home Assistant seeding
- ❑ D1 configuration.yaml: legacy hand-staged block (bitcoind :18332 + plaintext RPC creds) fully replaced by the bounded token-based block.
- ❑ D2 `custom_sentences/en/archy.yaml` carries all four intents.
- ❑ D3 `.storage/core.config_entries`: wyoming entry for openwakeword (:10400) + `anthropic` entry (Claude, conversation + ai_task subentries).
- ❑ D4 Pipeline: `conversation_engine = conversation.claude_conversation`, `prefer_local_intents: true`.
- ❑ D5 automations.yaml: `archy_mesh_announce` seeded.
- ❑ D6 HA restarts clean — no setup errors for anthropic / wyoming / rest / intent_script in `podman logs homeassistant`.
- ❑ D7 Sensors report real values: archy_block_height, archy_bitcoin_sync, archy_bitcoin_peers, archy_mesh_peers, archy_lightning_balance (or clean unavailable if LND absent), archy_mesh_message.
## E. Voice / intents (API level first, then live speaker)
- ❑ E1 Exact phrase "what's the block height" → answered by the LOCAL intent (correct height, no Anthropic API call in HA logs).
- ❑ E2 Fuzzy phrase (e.g. "how tall is the chain right now") → Claude routes to the ArchyBlockHeight tool; answer contains the real height.
- ❑ E3 "how many peers", "is the node synced", "what's my lightning balance" → correct spoken-length answers.
- ❑ E4 Off-topic question → Claude answers, 1–2 sentences, no markdown.
- ❑ E5 **(user)** Live speaker: "Hey Jarvis, what's the block height" → audible correct answer.
- ❑ E6 Mesh announce: new received mesh text (or manual `assist_satellite.announce` if no radio) → speaker announces sender + text; no announce storm on HA restart.
## F. Pine launcher page (1.3.0)
- ❑ F1 Page on :10380→:10381 shows the live node card (version, uptime, block, sync, peers) within ~5s.
- ❑ F2 `/node-status` proxy works (pine nginx resolves host.containers.internal at startup — container must not crash-loop).
- ❑ F3 "Connect Pine to WiFi" provisioner still intact (no JS errors on load).
## G. Cleanup / regression sweep
- ❑ G1 Both stray socat 18332 forwarders killed; sensors still work via the endpoint.
- ❑ G2 No bitcoind RPC credentials anywhere in HA config.
- ❑ G3 Pre-existing HA function intact: whisper/piper entities, PineVoice satellite pairing, other integrations.
- ❑ G4 nginx regressions: `/health`, `/bitcoin-status`, `/api/app-catalog`, `/proxy/lnd/` all still proxied post-patch.
- ❑ G5 **(user)** Mobile Home: wallet card sits directly under My Apps; desktop layout unchanged.
- ❑ G6 Other agent's changes re-verified after merge (their own checklist).
## H. Production-readiness (release ritual gate)
- ❑ H1 `cargo test` workspace green; frontend builds; drift check `--release --strict` green.
- ❑ H2 `tests/lifecycle/run-gate.sh` re-run ON .228 (stack membership changed → lifecycle gate rule applies).
- ❑ H3 Catalog regenerated → signed (ceremony) → published via gitea-ai; verified at the raw URL.
- ❑ H4 Changelog (layman-readable) + `scripts/sync-whats-new.py` + version bump; release ritual per v1.7.110 notes (push main via gitea-ai BEFORE publish; sign manifest AFTER create-release).
- ❑ H5 No secrets in any commit; frontend tarball flat + APK policy per release notes.
docs/qr-scanner-snappiness-handover.md
-82
View File
@@ -1,82 +0,0 @@
# QR scanner snappiness — research + companion-dev handover
*2026-07-29. Owner: web side = node repo (this doc's "web" items); native side =
companion app dev (Mac). Backlog origin: UNIFIED-TASK-TRACKER "optimise
companion QR scan (quicker start/decode, low-light)".*
## Where scanning happens today
| Path | Stack | Used when |
|---|---|---|
| Web live scan | nimiq `qr-scanner` 1.4.x over `getUserMedia`, in `WalletScanModal.vue` | HTTPS browsers / secure contexts |
| Photo fallback | `<input capture>` photo → `BarcodeDetector` if present, else `qr-scanner.scanImage` multi-pass (`decodePhotoRobust`) | Plain-http (LAN) where `getUserMedia` doesn't exist |
| Native scan | `ArchipelagoQr` JS bridge → companion's native scanner (0.5.22 fixed dense invoice QRs) | Inside the companion app |
## What makes it feel slow (ranked)
1. **Camera cold-start** — the stream starts only after the user reaches the
scan pane; on phones `getUserMedia` + first frame is routinely 600–1500ms,
and the native path pays a similar CameraX bind + ML Kit model cold-start.
2. **Decode cadence** — web live scan was capped at 4 scans/sec (WebView
preview lagged at 10/s when decoding on the JS worker). A hand-held code
therefore waits up to 250ms *after* it's already sharp and centered.
3. **Low light / focus hunting** — no torch control anywhere; no explicit
continuous-focus request. Dense LN invoices need sharpness more than
resolution.
4. **Dense-QR decode budget** — big bolt11/catalog QRs push the JS decoder
hard; the native ML Kit path is far better at these (proven by 0.5.22).
## Web side (node repo — can be done here)
- ✅ DONE (2026-07-29): scan at **10/s when `BarcodeDetector` exists** (Chrome/
Android WebView decode natively — cheap), keep 4/s only for the JS-worker
fallback.
- **Pre-warm the camera**: start `getUserMedia` the moment the modal opens
(action pane), not when the scan pane is reached — hide the preview until
needed. Saves the entire cold-start from the user's perceived timeline.
- **Torch toggle**: `qr-scanner` exposes `hasFlash()/turnFlashOn()` — add a 🔦
button on the scan pane (it silently no-ops where unsupported).
- **Continuous focus + modest resolution**: pass constraints
`{ focusMode: 'continuous', width: { ideal: 1280 } }` — 720p-class frames
start faster AND decode faster than 1080p+, with no loss for QR density
that matters to us.
- **Don't stop/start between panes**: returning from amount → scan currently
re-inits the scanner; keep the (paused) stream alive while the modal lives.
## Native side (companion dev handover)
The `ArchipelagoQr` bridge overlay is the right architecture — these are
tuning items inside the native scanner activity:
1. **Pre-warm CameraX + ML Kit**: bind the camera provider and instantiate
`BarcodeScanning.getClient(...)` when the WebView *requests* the overlay —
or even when the wallet modal opens (add a `ArchipelagoQr.prewarm()` bridge
method; the web side will call it if present). ML Kit's first-inference
model load is 100–300ms — pay it before the user aims.
2. **Restrict formats**: `BarcodeScannerOptions` with `FORMAT_QR_CODE` only —
skipping the other symbologies measurably cuts per-frame latency.
3. **Analysis resolution ≈ 1280×720** with
`ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST` — never queue stale frames;
decode the newest one only.
4. **Continuous autofocus + tap-to-focus** on the preview, and a **torch
toggle** (low-light was an explicit user complaint).
5. **`zoomRatio` nudge for small codes**: if no hit after ~2s, step zoom to
1.5× — helps distant/small printed codes without user action.
6. **Success haptic + instant dismiss**: vibrate on decode and close the
overlay immediately; perceived speed is heavily back-loaded.
7. Optional: **ML Kit `enableAllPotentialBarcodes` off** and skip inverted
scans unless first pass fails (inverted QRs are rare; halves work).
## Acceptance criteria
- Cold open → first successful scan of a normal invoice QR in **< 2s** on the
companion app, **< 3s** in a mobile browser.
- Dense (700+ char) bolt11 QR decodes in **< 1.5s** once framed, both paths.
- Dim-room scan succeeds with the torch toggle without leaving the scanner.
## Verification notes for whoever implements
- Measure with a timestamp log: overlay-requested → camera-first-frame →
decode-success. The three deltas map 1:1 onto items above.
- Web `BarcodeDetector` presence differs per WebView/Play-Services build —
keep the JS-worker fallback path intact.
docs/quadlet-compilation.md
+126
View File
@@ -0,0 +1,126 @@
# Manifest → Quadlet unit
How an app manifest becomes a Podman [Quadlet](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html)
`.container` unit that systemd owns, where the unit lands, and how to inspect one.
Source of truth: `core/archipelago/src/container/quadlet.rs`.
## Why Quadlet
Containers used to be fire-and-forget `tokio::spawn` blocks. If the daemon
crashed mid-spawn or the kernel reaped a parent cgroup, the container vanished
from `podman ps` and only a manual `podman run` brought it back. Quadlet removes
that whole class of failure: the unit lives on disk, **systemd owns
start/restart, and archipelago is just the provisioner**. This is the path that
runs the companion UI containers today (`archy-bitcoin-ui`, `archy-lnd-ui`,
`archy-electrs-ui`), and the validated path being flipped to default for apps.
## What gets generated
`Quadlet::from_manifest(manifest, name)` translates a manifest into a unit, and
`render()` produces the file. Every unit carries a header making clear it is not
hand-edited:
```ini
# Generated by archipelago. DO NOT EDIT.
# Edits are overwritten on the next reconcile.
[Unit]
Description=<app description>
After=network-online.target
Wants=network-online.target
Requires=<dependency>.service # one per declared dependency
After=<dependency>.service
[Container]
ContainerName=<name>
Image=<image ref>
Pull=never # image must be present locally already
Network=<host | pasta | slirp4netns | bridge name>
User=<uid> # when the manifest pins one
DropCapability=ALL # security default
AddCapability=<cap> # only capabilities the manifest opts into
PublishPort=<bind>:<host>:<container>/<proto>
Environment=<KEY>=<value> # non-secret env only
Secret=<secret_name>,type=env,target=<KEY> # secrets by REFERENCE, never value
Volume=<source>:<target><opts>
ReadOnly=true # when security.readonly_root
NoNewPrivileges=true # when security.no_new_privileges
HealthCmd=<cmd> # from the health_check block
[Service]
TimeoutStartSec=0
Restart=<always | on-failure> # from the restart policy
RestartSec=10 # 10s backoff caps a crash loop
[Install]
WantedBy=default.target
```
Two things to note in that mapping:
- **Secrets go in by reference, never by value.** A `secret_env` entry renders as
`Secret=<name>,type=env,target=<KEY>`, so podman injects the value at run time
from the node's secret store. The plaintext never appears in the unit file. See
[App secrets](secrets.md).
- **`Pull=never` is deliberate.** The provisioner does not pull images from here;
the image must already be local (pre-pulled or built). A missing image surfaces
immediately instead of retrying silently behind systemd's restart loop.
- **`PublishPort` is dropped entirely under `Network=host`.** Podman rejects the
combination and the container crash-loops on exit 125, so declared ports are
omitted rather than rendered. With host networking the container is already on
the host's ports; a manifest that declares both is not an error, the mapping is
just silently unnecessary.
## Where units land
Rootless, per-user, under the archipelago service user (uid 1000, with linger
enabled so the units run without an active login):
```
~/.config/containers/systemd/<name>.container
```
Quadlet's systemd generator translates `<name>.container` into a
`<name>.service` unit at **daemon-reload** time. Everything is `systemctl --user`
— the system bus is never touched from this path.
## Lifecycle: render → write → enable → disable
The module does four things and nothing else:
1. **render** — manifest → unit text (above).
2. **write** — `tempfile + rename` so a partially-written unit is never visible to
systemd, and `write_if_changed` compares bytes first: if the rendered unit
matches what is on disk, nothing is touched — no daemon-reload, no restart
cascade. This is what makes a reconcile tick cheap and non-disruptive.
3. **enable** — `daemon-reload` then start the `.service`.
4. **disable** — stop and remove.
## Inspecting a unit
Run these **as the archipelago service user** (the units are in its user bus):
```bash
# the generated unit
cat ~/.config/containers/systemd/archy-bitcoin-ui.container
# what systemd made of it
systemctl --user cat archy-bitcoin-ui.service
systemctl --user status archy-bitcoin-ui.service
journalctl --user -u archy-bitcoin-ui.service
# after editing a unit by hand for debugging (it will be overwritten on reconcile)
systemctl --user daemon-reload
```
Because the unit is regenerated on every reconcile, the way to change a
container's shape is to change its **manifest** (and, for a catalog-covered app,
regenerate and re-sign the catalog), never to edit the `.container` file — the
`DO NOT EDIT` header is literal.
## Related
- [Container lifecycle](container-lifecycle.md) — the reconciler that drives this
- [App Manifest Specification](app-manifest-spec.md) — the manifest fields mapped above
- [App secrets](secrets.md) — how `Secret=` references resolve
- [ADR-001: Podman over Docker](adr/001-podman-over-docker.md)
docs/registry-manifest-design.md
+25 -10
View File
@@ -9,18 +9,30 @@ signed app-catalog on the registry — **no OS-level code reliance, no
OTA-shipped disk manifest required**. Rootless, signed, robust, reboot-survivable.
See also: [`docs/dht-distribution-design.md`](dht-distribution-design.md) (this is
its "discovery/authenticity" layer), `MEMORY → project_manifest_driven_north_star`.
its "discovery/authenticity" layer).
---
## 1. Where we are today
## 1. Where we started (the pre-Phase-1 baseline)
Two distinct mechanisms, only one of which is registry-distributed:
This section is the problem statement the design was written against, kept for
context. **It no longer describes the running system** — Phases 1–3 shipped, so
see "Where we are now" below.
| Thing | Source | Reaches node via | Carries |
Two distinct mechanisms, only one of which was registry-distributed:
| Thing | Source | Reaches node via | Carried |
|-------|--------|------------------|---------|
| `apps/*/manifest.yml` (48) | repo working tree | **OTA**: `self-update.sh` rsyncs `apps/ → /opt/archipelago/apps/` | full manifest (the orchestrator's real source of truth) |
| `app-catalog.json` (28) | `releases/app-catalog.json` | **registry HTTP fetch**, hourly, **signed** (`app_catalog::refresh_catalog`) | version + image override only |
| `apps/*/manifest.yml` | repo working tree | **OTA**: `self-update.sh` rsyncs `apps/ → /opt/archipelago/apps/` | full manifest (the orchestrator's real source of truth) |
| `app-catalog.json` | `releases/app-catalog.json` | **registry HTTP fetch**, hourly, **signed** (`app_catalog::refresh_catalog`) | version + image override only |
### Where we are now
`releases/app-catalog.json` carries 66 entries, and 56 of them embed a full
`manifest` block — one for every `apps/*/manifest.yml` in the tree. So the
"catalog carries an image override only" gap below is closed for image-only
apps; what remains is build-context apps (Phase 4) and dropping `apps/` from the
OTA rsync (Phase 5).
- Orchestrator registry = in-memory `state.manifests: HashMap<app_id, LoadedManifest>`,
populated by `ProdContainerOrchestrator::load_manifests()` walking the disk dir.
@@ -44,7 +56,8 @@ binary OTA, no disk manifest.
publisher: apps/*/manifest.yml ──generate──▶ releases/app-catalog.json (embeds + signs)
node: refresh_catalog() ──fetch+verify──▶ <data_dir>/app-catalog.json
load_manifests() ──merge──▶ state.manifests (catalog wins; disk = fallback)
install(app_id) ──▶ render Quadlet unit (rootless, systemd-managed)
install(app_id) ──▶ create the rootless container (Quadlet unit when
use_quadlet_backends is on; podman create+start otherwise)
```
## 3. Schema change (`app_catalog::AppCatalogEntry`)
@@ -143,6 +156,8 @@ Add a generator (extend `create-release.sh` / a small `scripts/gen-app-catalog`)
separate signed blob? Inline is simplest for Phase 1; hashing aligns with the
DHT image-by-digest plan and keeps the catalog small. Lean inline now, revisit
at Phase 4 when build contexts (large) need addressing anyway.
- `generated_files` with inline content (vs. source-dir) — already supported in the
manifest schema? If so, registry manifests can carry small rendered files inline,
removing another disk dependency.
- ~~`generated_files` with inline content (vs. source-dir) — already supported in
the manifest schema?~~ **Answered: yes.** `app.files[]` takes inline `content`
(with `{{HOST_IP}}` / `{{NETWORK_GATEWAY}}` / `{{secret:NAME}}` rendering), so
registry manifests already carry small rendered files inline and that disk
dependency is gone. See [`app-manifest-spec.md`](app-manifest-spec.md).
docs/secrets.md
+117
View File
@@ -0,0 +1,117 @@
# App secrets
How an app declares a secret, how Archipelago materialises it, and how it
reaches the container — with the rules a developer must not break.
The whole point: **an app never ships a credential.** It declares the *shape* of
the secrets it needs, and the node generates a fresh, per-install value that
never leaves the node and is never logged. Source of truth:
`core/archipelago/src/container/secrets.rs` and the manifest schema in
`core/container/src/manifest.rs`.
## The two halves
A secret has a producer and a consumer, and they are separate manifest fields:
- **`generated_secrets`** — *produce* a random value into a file.
- **`secret_env`** — *inject* a file's contents into the container as an env var.
An app can use either alone. A generated secret with no consumer is just a file
on the node; a `secret_env` with no matching `generated_secrets` reads a file
that some other component (or the daemon) is expected to have written.
## Declaring a generated secret
```yaml
container:
generated_secrets:
- name: btcpay-db-password
kind: hex16
- name: fedimint-gateway-hash
kind: bcrypt
```
`name` is a **bare filename** under the node's secrets directory
(`/var/lib/archipelago/secrets/`). It is validated at manifest-load time — no
`/`, no `..` — so a manifest cannot write outside that directory.
`kind` chooses how the value is produced. Each kind is deterministic in *shape*
(the orchestrator knows exactly which files it will create) but random in value:
| `kind` | Value | Files written | Use for |
|---------|-----------------------------------------|-----------------------------------|---------|
| `hex16` | 16 random bytes, lowercase hex (32 ch) | `<name>` | service passwords, API tokens |
| `hex32` | 32 random bytes, lowercase hex (64 ch) | `<name>` | longer keys/cookies |
| `base64`| 32 random bytes, standard base64 (44 ch)| `<name>` | services that base64-decode their key (e.g. netbird relay `authSecret`) |
| `bcrypt`| a random password **and** its bcrypt hash| `<name>` (hash) + `<name>.pw` (plaintext) | server configured with a hash, client needs the plaintext |
`bcrypt` is the only kind that writes two files: `<name>` holds the bcrypt hash a
server is configured with, and `<name>.pw` holds the plaintext for any client
that must authenticate against it. A `secret_env` injects whichever of the two it
references.
## Injecting a secret into the container
```yaml
container:
secret_env:
- key: BTCPAY_DB_PASS
secret_file: btcpay-db-password
```
At apply time the orchestrator reads `/var/lib/archipelago/secrets/<secret_file>`
and makes it available in the container as `<key>`. It does **not** do this by
adding `KEY=value` to the environment — that value would show up in
`podman inspect` output and, on the Quadlet path, as a plaintext `Environment=`
line in a unit file on disk. Instead the resolved pairs are registered as podman
secrets named `archy-env-<app-id>-<key>` and referenced by name, so the value
never lands in the manifest, a unit file, `podman inspect`, or a log line.
**Interpolation taints.** A plain `environment` entry that interpolates a secret
— e.g. BTCPay's `ConnectionString=...Password=${BTCPAY_DB_PASS}` — is treated as
secret-bearing itself and travels the same protected path, rather than being
left in the clear because it was declared under `environment`. So you can build
connection strings from secrets without leaking them.
## How materialisation works
`ensure_generated_secrets()` runs on **every install and reconcile tick**, before
`secret_env` is resolved. It is idempotent and self-healing:
1. **Fast path.** If every target file for a secret already exists, is readable
by the service user, and is non-empty, it is left untouched. A secret is
generated **once** and then persists across restarts, updates and reinstalls —
this is what makes credentials stable (migrations never regenerate a working
secret out from under a database).
2. **Self-heal.** A target file that exists but is unreadable or empty — e.g.
left root-owned by a botched earlier write — is removed and recreated, owned
by the service user. The unlink uses the secrets directory's own write bit, so
recovery needs no privilege escalation.
3. **Write.** New values are written through an atomic `0600` writer: a temp file
in the same directory, fsynced, then renamed over the target, so a reader never
sees a half-written secret and the file is only ever readable by its owner.
Because it runs every tick and no-ops when the secret is healthy, calling it is
always safe; there is no separate "provision secrets" step to forget.
## Rules a developer must not break
- **Never hardcode a credential**, in the manifest or in code, even as a
fallback. A shared fallback password means everyone holding a copy of the repo
holds that credential. Declare `generated_secrets` instead.
- **Never log a secret.** `secret_env` values and the files under the secrets
directory stay out of logs, error messages and status output.
- **One canonical name.** The orchestrator, first-boot script, reconcile path and
any deploy tooling must all reference a secret by the *same* filename. A
producer writing `<app>-password` while the consumer reads `<app>-hash` yields a
service that authenticates against a credential nothing generated.
- **Pick the encoding the service expects.** `hex*` and `base64` decode to
different bytes; a service that base64-decodes its configured key must be given
a `base64` secret, or it will run with the wrong key material.
## Related
- [App Manifest Specification](app-manifest-spec.md) — the full manifest schema
- [ADR-009: Manifest-Level Container Security](adr/009-manifest-container-security.md)
- [Entropy Enforcement (KEY-05)](security/KEY-05-ENTROPY-ENFORCEMENT.md) — why secret
generation draws from an explicitly-named CSPRNG
docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md
+19 -10
View File
@@ -1,7 +1,7 @@
# The Bitcoin RPC proxy that stayed open after it was fixed
**Status:** code fix committed (`f6b5245b`); on-node verification recorded below.
**Found:** 2026-08-02, archi-dev-box, while verifying `a05956c4` instead of assuming it.
**Found:** 2026-08-02, a test node, while verifying `a05956c4` instead of assuming it.
**Severity:** critical on any affected node — unauthenticated control of Bitcoin Core RPC
through a proxy that injects the node's own credentials.
@@ -25,9 +25,9 @@ cookies, no credentials:
| Probe | Result |
|---|---|
| `GET http://192.168.63.240:18083/lnd-connect-info` | `401`, 24 bytes, `{"error":"Unauthorized"}` — **closed** |
| `POST http://192.168.63.240:8334/bitcoin-rpc/` (`getblockcount`) | `200` — `{"result":960774,"error":null}` — **OPEN** |
| `OPTIONS http://192.168.63.240:8334/bitcoin-rpc/` | `204` with `Access-Control-Allow-Origin: *` — **OPEN** |
| `GET http://192.0.2.240:18083/lnd-connect-info` | `401`, 24 bytes, `{"error":"Unauthorized"}` — **closed** |
| `POST http://192.0.2.240:8334/bitcoin-rpc/` (`getblockcount`) | `200` — `{"result":960774,"error":null}` — **OPEN** |
| `OPTIONS http://192.0.2.240:8334/bitcoin-rpc/` | `204` with `Access-Control-Allow-Origin: *` — **OPEN** |
The rendered config on disk, `/var/lib/archipelago/bitcoin-ui/nginx.conf`, was dated
**2026-06-30** — the pre-fix version, with no `auth_request` and with the wildcard CORS
@@ -79,7 +79,7 @@ Deliberately narrow:
survived. A new regression test pins the whole chain: stale conf in, gate present out,
container restarted, nothing created.
## What actually closed it on archi-dev-box — and what that does NOT prove
## What actually closed it on a test node — and what that does NOT prove
Sequence, from file mtimes, container start times and the daemon journal:
@@ -88,7 +88,7 @@ Sequence, from file mtimes, container start times and the daemon journal:
| 18:33 | Probe: `POST /bitcoin-rpc/` → `200` with a real block height. Exposure confirmed live. |
| 18:36 | A **separate rebuild of bitcoin-ui**, done outside this work, rendered the fixed conf and recreated `archy-bitcoin-ui`. `:8334` closes here. |
| 19:06 | The binary carrying `f6b5245b` is installed and the daemon restarted. |
| 19:12 | Probe: `POST /bitcoin-rpc/` → `401`. `OPTIONS` now returns `Access-Control-Allow-Origin: http://192.168.63.240:8334`, not `*`. |
| 19:12 | Probe: `POST /bitcoin-rpc/` → `401`. `OPTIONS` now returns `Access-Control-Allow-Origin: http://192.0.2.240:8334`, not `*`. |
So the node is closed, and the fixed template is proven to work end to end on real
hardware — but **the reconcile fix itself was never exercised.** By the time it was
@@ -104,7 +104,7 @@ Tracked as broken window 15 — **since closed by the controlled test below.**
## Proving the delivery path on real hardware
Run on archi-dev-box, 2026-08-02 20:00–20:03 EDT, with operator approval. The point was to
Run on a test node, 2026-08-02 20:00–20:03 EDT, with operator approval. The point was to
prove the thing the incidental rebuild had made unprovable: that **reconcile itself**
repairs this state, unaided.
@@ -118,7 +118,7 @@ which mechanism produced it.
| 2 | Probe with no cookies | `POST /bitcoin-rpc/` → **`200`**, `{"result":960790}`; `Allow-Origin: *`. **Genuinely re-exposed** |
| 3 | Start the daemon (20:00:36) and touch nothing further | — |
| 4 | Reconcile pass at **20:02:19** | `bitcoin_ui: nginx.conf rendered auth_hash=51f2b5af`, then `WARN prod_orchestrator: rewrote config for a user-uninstalled app whose container is still RUNNING (systemd/Quadlet keeps it alive independently of reconcile) — restarting so it picks the new config up app_id=bitcoin-ui container=archy-bitcoin-ui` |
| 5 | Probe again | `POST /bitcoin-rpc/` → **`401`**; `Allow-Origin: http://192.168.63.240:8334` |
| 5 | Probe again | `POST /bitcoin-rpc/` → **`401`**; `Allow-Origin: http://192.0.2.240:8334` |
| 6 | Compare state | Conf **byte-identical** to the pre-test known-good; container healthy |
Step 2 is what makes steps 4–6 mean anything: without a confirmed `200`, the later `401`
@@ -134,12 +134,21 @@ The operator's call, recorded here so it is not silently re-litigated: **no LND
rotation, and no Bitcoin RPC password rotation.** The reasoning was that there is no
evidence of exploitation and the vulnerability is being closed rather than lived with.
`scripts/security/rotate-lnd-macaroon.sh` stays in the tree as a tool. It has been
exercised in detect mode only, and has never rotated anything on any node. Its ordering
`scripts/security/rotate-lnd-macaroon.sh` stays in the tree as a tool. Its ordering
guard (refuses to rotate on a binary lacking the fix) remains the right shape for whenever
rotation is wanted — including for the Bitcoin RPC password, which has no equivalent tool
yet.
**Amended 2026-08-08.** This section said the script "has never rotated anything on any
node"; that is no longer true. A rotation was performed on a development node while
responding to the BTCPay Server advisory (that node had been running an affected
`btcpayserver:2.3.9`), and it exposed a gap the script did not cover: BTCPay's inline copy
of the macaroon was left stranded, so its Lightning payments failed silently while both
apps reported healthy. Rotation is now a first-class, password-confirmed dashboard action
that repairs that copy as part of the run — see
[`LND-MACAROON-ROTATION.md`](LND-MACAROON-ROTATION.md). The fleet decision recorded above
is unchanged: no fleet-wide rotation for this leak.
What this decision accepts: any macaroon or RPC password read through either hole before
it was closed stays valid. That is a deliberate, informed trade, not an oversight.
docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md
-1006
View File
File diff suppressed because it is too large Load Diff
docs/security/KEY-01-ON-NODE-VERIFICATION.md
-224
View File
@@ -1,224 +0,0 @@
# KEY-01 on-node verification — audit item C-6 and the F-01 refusal proof
**Status: INCOMPLETE — C-6 is NOT yet verified.**
**Opened:** 2026-08-02 · **Phase:** 10 (key-material hardening) · **Plan:** 10-02
**Probe:** `scripts/security/rpc-exposure-probe.sh`
This document records on-node evidence for
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §6 item **C-6** ("Is the RPC endpoint
reachable unauthenticated from the LAN?") and for the KEY-01 / F-01 refusal shipped by
plan 10-01 (`core/archipelago/src/api/rpc/onboarding_gate.rs`, commit `879de59e`).
Nothing below is recorded unless it was actually executed and its output observed. Rows
marked **NOT MEASURED** are open work, not assumptions. Per threat T-10-13 this document
records node **labels** and status codes only — never raw LAN addresses, onion addresses
or mesh ULAs, because this repository is being prepared for open-sourcing.
---
## Probe-method correction
**The audit's own C-6 command cannot detect the condition it claims to test. Do not
re-derive this; it has now been checked against the code twice.**
`ENTROPY-SEED-AUDIT-2026-07-31.md:890-901` probes with `seed.status` and declares
`200` a failure. But `seed.status` is **not** in `UNAUTHENTICATED_METHODS`
(`core/archipelago/src/api/rpc/middleware.rs:5-38`, which lists `seed.generate`,
`seed.verify`, `seed.restore` and `seed.save-encrypted` — not `seed.status`). An
unauthenticated `seed.status` is therefore rejected at
`core/archipelago/src/api/rpc/mod.rs:293` with a **401 by design**. The audit's "Fail:
200" criterion can never fire, so the probe would report the unauthenticated surface as
closed while F-01's actual door stands open.
`scripts/security/rpc-exposure-probe.sh` measures the two facts separately:
| Signal | Method | Why | Reading |
|---|---|---|---|
| **Exposure** | `auth.isOnboardingComplete` | genuinely unauthenticated (`middleware.rs:9`), read-only, no side effects | `200` = the unauthenticated RPC surface is reachable from this vantage point. This is the honest C-6 result. |
| **Session enforcement** | `seed.status` | deliberately *not* allowlisted | `401` = the session check is working. Anything else is a worse finding than C-6 and halts the phase. |
The probe reports a reachable unauthenticated surface as `EXPOSED`, not `FAIL`: on the LAN
this is the current expected posture, and the purpose of C-6 is to **measure** the surface,
not to assert it is already closed.
---
## C-6 — unauthenticated RPC reachability
### Result table
| Transport | Label | `health` | `auth.isOnboardingComplete` (exposure) | `seed.status` (enforcement) | Status |
|---|---|---|---|---|---|
| Loopback | `loopback` | 200 | **200 — EXPOSED** | **401 — PASS** | measured 2026-08-02 |
| Node's own LAN address, probed *from the node itself* | `self-lan-ip` | 200 | **200 — EXPOSED** | **401 — PASS** | measured 2026-08-02 |
| LAN, from a second machine | `lan` | — | — | — | **NOT MEASURED** |
| Tor onion | `tor` | — | — | — | **NOT MEASURED** |
| FIPS mesh ULA, from a peer node | `mesh` | — | — | — | **NOT MEASURED** |
**`seed.status` returned `401` on every vantage point actually tested.** No
stop-the-plan condition was observed.
### Why the two measured rows are NOT a C-6 result
Both runs originated **on the node under test**. Packets to the node's own addresses are
delivered by the local stack and never traverse the LAN, so neither run exercises the
external path an attacker would use, and neither run passes through any host or upstream
filtering that applies only to foreign packets. They are recorded because they establish
two real facts — the probe works against a live daemon, and session enforcement is intact
— but C-6 asks specifically whether a **different machine** can reach the surface, and
that question is still open.
### Verbatim probe output (measured rows)
```
$ bash scripts/security/rpc-exposure-probe.sh --target 127.0.0.1 --scheme http --port 80 --label loopback
RPC exposure probe — label=loopback endpoint=http://127.0.0.1:80
audit item C-6 · KEY-01 (F-01) · read-only mode
[loopback] health 200 REACHABLE endpoint answers from this vantage point
[loopback] auth.isOnboardingComplete 200 EXPOSED unauthenticated RPC surface IS reachable from here (C-6 result)
[loopback] seed.status 401 PASS session enforcement active for non-allowlisted methods
[loopback] auth.isOnboardingComplete (/rpc/) 404 NOT-EXPOSED alternate proxy path did not answer 200
exit=0
```
```
$ bash scripts/security/rpc-exposure-probe.sh --target <node-lan-ip> --scheme http --port 80 --label self-lan-ip
RPC exposure probe — label=self-lan-ip endpoint=http://<node-lan-ip>:80
audit item C-6 · KEY-01 (F-01) · read-only mode
[self-lan-ip] health 200 REACHABLE endpoint answers from this vantage point
[self-lan-ip] auth.isOnboardingComplete 200 EXPOSED unauthenticated RPC surface IS reachable from here (C-6 result)
[self-lan-ip] seed.status 401 PASS session enforcement active for non-allowlisted methods
[self-lan-ip] auth.isOnboardingComplete (/rpc/) 404 NOT-EXPOSED alternate proxy path did not answer 200
exit=0
```
### Corroborating host state (observed, but NOT a substitute for the LAN measurement)
Recorded because it predicts the LAN result and tells the operator what to expect:
- nginx listens on **`0.0.0.0:80` and `[::]:80`** (`ss -ltn`), i.e. on every interface,
not on loopback only. The daemon itself is bound loopback-only on `127.0.0.1:5678`, so
all external reachability is via nginx.
- The host packet filter does **not** block port 80: `iptables -S INPUT` is
`-P INPUT ACCEPT` with a single jump into Tailscale's chain, and the `nft` ruleset
contains only Tailscale's `ts-input`/`ts-forward` chains — no rule matching tcp/80.
Together these make an `EXPOSED` LAN result very likely. **That is a prediction, not a
measurement, and C-6 stays open until a second machine produces the status code.**
### Incidental finding — `/rpc/` is not a second door
`auth.isOnboardingComplete` on nginx's `location /rpc/` block
(`image-recipe/configs/nginx-archipelago.conf:192`) returned **404** from both vantage
points. The block proxies the full URI to the backend, which only routes `/rpc/v1`, so
the unauthenticated surface is reachable through exactly one path. This narrows F-01's
exposure surface by one path and should be re-checked if the nginx config changes.
---
## KEY-01 refusal check — NOT PERFORMED
**Requirement:** on a node running 10-01's gate, an unauthenticated `seed.restore`
carrying attacker-supplied words is refused, and `identity/node_key` and
`identity/nostr_secret` are byte-identical afterwards.
**Blocker — no node in the fleet is running 10-01's gate yet.** Verified on the dev-box
rather than assumed:
```
$ ls -l /usr/local/bin/archipelago
-rwxr-xr-x 1 root root 53437536 Aug 2 06:37 /usr/local/bin/archipelago
$ git log -1 --format='%H %ci' 879de59e
879de59eccb489d590c8e0fca6ae79098df68200 2026-08-02 13:05:35 -0400
$ grep -qa "Not supported: this node is already provisioned" /usr/local/bin/archipelago \
&& echo PRESENT || echo ABSENT
ABSENT
```
The installed binary was built at 06:37; 10-01 landed at 13:05 the same day, and the
gate's refusal string is absent from the running binary. A `--destructive` run against
this node would therefore **not** be refused — it would replace the node's identity. The
dev-box is a live dev-pair deploy target in real use, so the run was not made.
**This check is blocked on deployment, which the phase brief explicitly excludes from
this plan.** It cannot be closed by any amount of work inside the repository.
---
## Fresh-node onboarding non-regression — NOT PERFORMED
**Requirement:** a genuinely un-onboarded instance completes the whole wizard with 10-01's
gate in place (the anti-brick proof for correctness trap 1 and the D-03a signal
correction), then refuses `seed.restore` immediately afterwards.
**Blocker — no un-onboarded instance exists.** The intended harness is shape (A) of
`.planning/todos/pending/2026-08-01-archi-dev-box-as-fresh-test-node-without-iso.md`
(a second daemon under its own `ARCHIPELAGO_DATA_DIR`/`ARCHIPELAGO_BIND`/
`ARCHIPELAGO_PORT_OFFSET`), and that todo is still **pending** — the harness has not been
built. It would additionally need a binary built from `879de59e` or later, which the
running daemon is not.
Note for whoever builds it: that todo records that several constants ignore
`ARCHIPELAGO_DATA_DIR` and point at `/var/lib/archipelago` literally
(`bitcoin_rpc.rs:10`, `container/lnd.rs:131`, `electrs_status.rs:15`,
`api/rpc/package/pine_ha.rs:34-36`, `bootstrap.rs:242`, `disk_monitor.rs:41`), so a shape-A
instance must not install Bitcoin, LND, electrumx or Pine/HA — it would read and write the
live node's files. The onboarding walkthrough this check needs does not install apps, so
the hazard is avoidable, not blocking.
---
## Pre-OTA fleet check carried over from 10-01
10-01's summary records a state that its gate makes unrecoverable: a node with
`onboarding.json = {"complete": true}` but **no** `user.json` can no longer call
`auth.setup`, and the recovery path needs a session it cannot create. Recovery is one SSH
command (`rm /var/lib/archipelago/onboarding.json`), but the fleet must be checked
**before** the OTA ships (D-10).
| Node label | `user.json` | `onboarding.json` | Verdict |
|---|---|---|---|
| dev-box | PRESENT | `{"complete": true}` | **safe** — provisioned normally; the gate refuses re-keying, which is the intent |
| rest of fleet | — | — | **NOT CHECKED** |
Command to run per node:
```bash
ls -l /var/lib/archipelago/user.json /var/lib/archipelago/onboarding.json 2>&1
cat /var/lib/archipelago/onboarding.json 2>/dev/null
```
A node is at risk only if `onboarding.json` says `complete: true` **and** `user.json` is
absent.
---
## What is still required to close C-6 and KEY-01
Every item below needs an operator with fleet access; none can be done from the repository.
1. **LAN exposure.** From a second machine on the node's LAN:
`bash scripts/security/rpc-exposure-probe.sh --target <node-lan-ip> --scheme http --port 80 --label lan`
2. **Tor exposure.** `torsocks bash scripts/security/rpc-exposure-probe.sh --target <onion> --scheme http --port 80 --label tor`
3. **Mesh exposure.** From a peer node over the FIPS mesh ULA:
`bash scripts/security/rpc-exposure-probe.sh --target <fips-ula> --scheme http --port 80 --label mesh`
(the peer listener allows `/rpc/v1` — `core/archipelago/src/server.rs:1270-1296` — so a
`200` confirms the mesh half of F-01's reachability claim). An unreachable transport is
recorded as `UNREACHABLE` with its error, never omitted.
4. **Deploy 10-01 to a disposable node**, then, from a second machine:
`bash scripts/security/rpc-exposure-probe.sh --target <disposable-node> --destructive --label refusal`
with `sudo sha256sum /var/lib/archipelago/identity/node_key /var/lib/archipelago/identity/nostr_secret`
captured on the node immediately before and after. The response must carry the
`Not supported:` prefix and the two digests must match character for character.
5. **Build shape (A)** and walk the wizard end to end on a 10-01 binary
(intro → options → path → seed → seed-verify → did → identity → backup → verify → done,
then set the password), reloading once on the seed screen to confirm the same 24 words
return. No `Not supported:` and no `Rate limit exceeded` may appear at any point. Then
re-run step 4 against that same instance to confirm the door closed behind onboarding.
6. **Check the remaining fleet** for the `onboarding.json`-without-`user.json` state above.
Until items 1–3 are done, audit item **C-6 remains UNVERIFIED**. Until item 4 is done, the
KEY-01 refusal is proven only by 10-01's unit tests against temp directories, never against
a running daemon over HTTP.
docs/security/KEY-02-FLEET-ROTATION.md
-244
View File
@@ -1,244 +0,0 @@
# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)
Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
build half (10-03).
10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
TLS private key that every downloader of that ISO also holds, and it will never try again.
This document records the two human decisions that govern the deployed half.
---
## D-06 rotation trigger
**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.
Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
flag").
### Why
Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
machine that has ever connected to it, and the old private key is destroyed by the swap. The
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
node needs physical console access to recover, which for `.228` means a site visit.
Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
of nodes still to rotate is a fact on a screen rather than a guess.
This also matches the project's standing policy that changes are verified on the dev pair
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
has been observed on the dev pair it has already run everywhere.
### What this decision binds
- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped
invocation is inert.
- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
It contains no apply path. Making the boot unit rotate would require editing the unit, which is
a deliberate act, not a default.
- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
the guard against "operator runs it on the wrong node" is structural, not procedural.
### Consequence recorded honestly
Any node whose verdict comes back `shared` and which is never revisited stays exposed
indefinitely. The mitigation is the visibility, not the automation. The list under
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.
---
## How a node decides
Four on-disk signals, evaluated in this precedence order by
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
that produced it, and each evidence string names the file it was read from.
| # | Signal | Source |
|---|---|---|
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` |
Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`.
**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the
verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather
than `per-node` — the node's own generator most recently reported failure, so a clean-looking
mtime is not evidence of success.
Signal 4 changes the meaning of missing material rather than adding to the shared/per-node
question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an
absent host key is a **fail-closed** state (generation never succeeded), not a shared one.
---
## C-3 — per-node host key and TLS uniqueness
Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described
there as "the highest-value check here".
### Status: **FAILED — with finding.** Recorded 2026-08-02.
> **This section names live fleet nodes that are still running shared key material.
> Review it before this repository is made public** (`docs/OPEN-SOURCE-READINESS-PLAN.md`).
> Digests below are truncated; the fingerprints of public keys are public data — every SSH
> handshake offers them — but there is no reason to make a target list convenient.
**Three distinct live fleet nodes share all three of their SSH host keys. Two of those three
also share their TLS certificate, and therefore their TLS private key.** This is not a
theoretical exposure: it is F-03 in production, today.
#### Method
Gathered **remotely and read-only** — no node was logged into, nothing was written to any node,
nothing was rotated. Host keys came from `ssh-keyscan`, which is what every SSH client does
before it decides whether to trust a host, and certificates from an anonymous TLS handshake:
```bash
ssh-keyscan -T 6 <node> | ssh-keygen -lf -
openssl s_client -connect <node>:443 </dev/null 2>/dev/null \
| openssl x509 -noout -fingerprint -sha256 -subject
```
This is a deliberately weaker instrument than the checklist's on-node commands, and it was chosen
because it needs no access and can therefore cover the whole reachable fleet rather than two
nodes. What it can prove is exactly the FAIL condition: *any fingerprint appearing on two nodes*.
#### Result
| Node label | SSH host keys (ECDSA/ED25519/RSA, truncated) | TLS cert sha256 (truncated) | Cert CN |
|---|---|---|---|
| `archipelago-1` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archy-x250-beta` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
| `archipelago` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `7C:6B:CD:98…` | `austin-sapien` |
| `archipelago-5` | `/bmgd6jS…` / `SpaNfLLf…` / `hhVFABi3…` | `95:FE:EB:C7…` | `archipelago.local` |
| `archi-dev-box` | `8hFU7QGM…` / `GAxNAcgX…` / `Tv7AfaVp…` | (no :443 listener) | — |
| `archy-dev-pa` | `JtD/RM0a…` / `XD2A5OVL…` / `esIBpbWk…` | not probed | — |
| `framework-pt` | `oicpsj3Y…` / `zxA1/kRU…` / `oxi+tMli…` | `88:85:CE:CC…` | `framework-pt` |
| `shorty-s` (`.228`) | `YVsgrv8M…` / `D/5n851i…` / `YMFLUerk…` | `4D:98:D4:9B…` | `shorty-s` |
Unreachable at scan time, so **UNVERIFIED**: `archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`,
`quantumterminal`.
#### That the three are genuinely different machines, not one host seen three times
The obvious alternative explanation for identical host keys is a single machine registered on the
tailnet more than once. Ruled out:
- All three answered a live TCP connection on port 22 within the same minute. One `tailscaled`
instance serves one tailnet identity, so three simultaneously-live addresses are three hosts.
- `tailscale ping` resolves them to **different physical endpoints**: `archy-x250-beta` answers
from `178.38.147.13` (and over the Frankfurt DERP), while `archipelago-1` and `archipelago`
answer from `45.20.199.86` on different source ports — a different continent for the first,
and two distinct machines behind one NAT for the other two.
- They are owned by different tailnet accounts.
#### Why `archipelago` has a different TLS cert but the same SSH keys
Its cert CN is `austin-sapien`, not the image default `archipelago`. That is the signature of a
node that was **renamed** through `server.set-name`, which re-mints the TLS cert via
`regenerate_tls_cert()` so the SAN matches the new hostname — and touches nothing else.
This is worth stating plainly because it is a trap: **TLS uniqueness alone is not evidence that
a node's key material is per-node.** Any renamed node gets a unique certificate for free while
its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on TLS
fingerprints only, `archipelago` would have looked clean. The SSH host key is the reliable
signal, and this is why the audit script treats the two classes separately and reports which one
is shared rather than issuing a single node-level verdict.
#### What this does NOT establish — UNVERIFIED
| Claim | Status | Evidence still needed |
|---|---|---|
| The three nodes were flashed from the **same ISO** | UNVERIFIED | Not required for the FAIL — shared host keys are the exposure however they got there — but the ISO build id would tell us how many other downloads carry the same keys. Needs on-node `/opt/archipelago/` provenance. |
| The audit script's verdict on those three nodes | UNVERIFIED | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on each. Requires the OTA carrying this plan's runtime payload to land, or the script to be hand-staged. Predicted `shared`; predicted is not observed. |
| A rotation preserves the operator's own session | UNVERIFIED **on hardware** | Checkpoint steps 4–6: run `--apply --yes` on one disposable node from a session you are willing to lose, confirm that session survives, confirm a second connection shows the expected mismatch. The harness proves the script's ordering and its abort path; it cannot prove that `systemctl reload ssh` keeps a real forked session alive. |
| `host_secrets` reaches `system.stats` on a real node | UNVERIFIED | Needs a build carrying this plan deployed to the dev pair, then a `system.stats` call. Proven in unit tests against the file contract only. |
| The four unreachable nodes | UNVERIFIED | Re-run the scan when they come back online. |
#### Consequence
`archipelago-1`, `archy-x250-beta` and `archipelago` are a **confirmed live F-03 instance**.
Anyone holding a copy of the ISO these nodes were flashed from holds their SSH host private keys,
and for the first two, their TLS private key as well — enough for undetectable SSH host
impersonation and transparent MITM of the web UI.
None of them was rotated as part of this verification, and that is deliberate: this checkpoint
verifies, it does not remediate, and remediating a node inside a verification task is how a
verification task takes a node offline. They are recorded below.
---
## Nodes with a `shared` verdict, deliberately not rotated
Any node that reports `shared` and is not rotated in the same session MUST be added here with the
date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible
list rather than an assumption.
| Node label | Date detected | Why not rotated | Next step |
|---|---|---|---|
| `archipelago-1` | 2026-08-02 | Detected by remote fingerprint comparison during C-3, not by an operator running the script. In real use; rotating it inside a verification task is exactly what the task forbids. | Stage the script, run `--detect`, then rotate from a session the operator is willing to lose. |
| `archy-x250-beta` | 2026-08-02 | Same. Also shares its **TLS private key** with `archipelago-1`, so it is the more urgent of the two. Reached over a DERP relay from another continent — the least recoverable node in the set if a rotation goes wrong. | Rotate from physical or console access if available; otherwise rotate TLS first, confirm, then SSH. |
| `archipelago` | 2026-08-02 | Same. TLS is already unique (the node was renamed, which re-mints the cert); only its SSH host keys are shared. | `--apply --yes` will rotate SSH only — the detect pass flags the classes separately, so this node's already-unique TLS pair is left alone. |
**Nobody has been told their `known_hosts` is about to break.** Three nodes here are in real use;
the rotation is one-way and every existing entry for them dies with it. Sequencing that is an
operator decision, which is the whole content of D-06.
---
## Operator runbook — rotating one node
Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or
any node in real use without arranging access recovery first.
```bash
# 1. Detect. Read-only; safe on any node, including production.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
cat /var/lib/archipelago/host-secrets-audit.json
# 2. Dry run. Prints the plan, touches nothing, exits 0.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply
# 3. Rotate. Only proceeds if the verdict is `shared`.
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes
# 4. WITHOUT closing that session, prove it survived:
echo still-here
# 5. From a second terminal, expect a host-key mismatch warning. That is the
# correct outcome. Update known_hosts against the fingerprints printed by
# step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
# blindly accepting whatever is offered.
ssh-keygen -R <node>
ssh <node>
# 6. The web UI will present a new self-signed cert. A fresh browser trust
# prompt is expected and is the correct outcome.
```
The script reloads sshd rather than restarting it. A reload re-execs the listener while
already-forked session children keep running, which is why the operator's own SSH session
survives its own rotation. `restart` would kill it, and on a remote node with no console that is
unrecoverable.
Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the
swap, so an operator who loses access anyway can still identify what changed.
docs/security/KEY-02-ROOTFS-EVIDENCE.md
-208
View File
@@ -1,208 +0,0 @@
# KEY-02 — build-host evidence for the rootfs identity strip
**Audit item:** C-4 of `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` (§868), which
belongs to finding **F-03** (fail-open, never-retried first-boot secret regeneration over
a fleet-shared rootfs).
**Status: ⛔ UNVERIFIED — awaiting a run on a real ISO build host.**
The code change is committed and unit-tested; the tar listing that proves its effect on a
real build has not been produced yet, because it requires a build host with podman/docker
and enough disk for a full rootfs rebuild. Do not read anything below the "Result" heading
as a passing check until it is filled in.
| Field | Value |
|---|---|
| Builder commit (Task 1) | `21043096` — fail-closed first-boot regeneration |
| Builder commit (Task 2) | `408b328c` — rootfs identity strip |
| Builder commit (follow-up) | single-producer unification, build-time generator assertion, self-heal timer |
| Builder file | `image-recipe/_archived/build-auto-installer-iso.sh` (LIVE; `image-recipe/build-debian-iso.sh` execs it) |
| Build host | _to be recorded_ |
| Date run | _to be recorded_ |
| RECIPE_HASH observed | _to be recorded — read it from the stamp file, see the caveat below_ |
---
## The expectation is deliberately INVERTED relative to the audit
This is the single most important thing to understand when comparing this document with the
audit, and the reason it is stated before the commands rather than after.
The audit's C-4 entry says:
> **Expected:** SSH host keys and the TLS key **present** (they are baked — see
> `build-auto-installer-iso.sh:345`, `:463-469`), `random-seed` **absent**, `machine-id`
> absent or zero-length. Anything else changes F-03's severity.
That expectation described the **broken** state the audit found, and recording it was how the
audit measured the size of F-03. Phase 10 plan 10-03 Task 2 removed that material. So:
**After this change, the audit's stated expectation is the FAILURE condition.** If SSH host
keys or the TLS private key still appear in the tar, the strip layer did not run — most
likely because a cached `archipelago-rootfs.tar` was reused. That is not a regression in the
check; it is the check working.
The two negative findings the audit recorded are unchanged and must still hold:
`var/lib/systemd/random-seed` absent, `etc/machine-id` absent or zero-length.
---
## Commands to run
Run all of these **on the build host**, from the repo root, on a checkout that contains
commits `21043096` and `408b328c`.
### 1. Force a full rebuild
The strip layer lives inside the `RECIPE_HASH` region (between the `# STEP 1: Build complete
root filesystem` and `# STEP 2: Build minimal installer` markers), so the hash changes and the
cached tar is invalidated automatically. `--rebuild` is passed anyway so that a stale tar
cannot mask the result for any reason:
```bash
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild
```
`UNBUNDLED=1` is mandatory per `CLAUDE.md` and project memory — the default env silently
builds the wrong full-bundle variant.
### 2. List the identity artefacts in the shipped tar
`WORK_DIR` is `image-recipe/build/auto-installer`, so:
```bash
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
| grep -E 'etc/ssh/ssh_host|etc/machine-id|var/lib/systemd/random-seed|archipelago/ssl/archipelago'
```
### 3. Expected result after this plan
- **no** `etc/ssh/ssh_host_*` entries at all
- **no** `etc/archipelago/ssl/archipelago.key` and **no** `archipelago.crt`
(the `etc/archipelago/ssl/` **directory** must still be present — the first-boot staging
swap needs somewhere to land)
- **no** `var/lib/systemd/random-seed`
- `etc/machine-id` present with size **0**, or absent. Either satisfies "not shared"; record
which one was actually observed rather than generalising.
Note on the TLS keypair specifically: it is now absent for two independent reasons, not one.
The Dockerfile no longer generates it at all (that layer was removed so there is a single
producer), *and* the strip layer still deletes it as belt-and-braces in case a future layer
starts baking one. Seeing it present therefore means both defences were bypassed.
### 4. Confirm the provenance file rode along
```bash
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar | grep rootfs-identity-stripped
```
Expected: one entry, `opt/archipelago/rootfs-identity-stripped`. Its absence means the strip
layer did not execute and the whole check is void.
### 5. Confirm the regeneration path and its self-heal timer are still shipped
This is the brick check, and it is not optional. A stripped rootfs whose first-boot
generation script failed to ship would leave every flashed node with no SSH host key and
nothing to create one. The timer is part of the same check: without it, a node whose
generators fail every in-boot retry has no unattended way back.
```bash
ls -l image-recipe/build/auto-installer/installer-iso/archipelago/scripts/first-boot-secrets.sh \
image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service \
image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.timer
```
Expected: all three present, `first-boot-secrets.sh` executable.
### 5b. Confirm the build-time generator assertion actually ran
The rootfs build fails outright if `openssl` or `ssh-keygen` is missing or non-executable,
because that is the one way first-boot generation can fail deterministically — retries and
reboots would never fix it, so it must never reach a node. A successful build therefore
already proves the generators are present, and the build log says so:
```bash
grep 'first-boot secret generators present' <build log>
```
If you did not capture the log, assert it against the tar instead:
```bash
tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
| grep -E 'usr/bin/(openssl|ssh-keygen)$'
```
Expected: both present and mode `-rwxr-xr-x`.
### 6. Record the RECIPE_HASH the builder actually used
```bash
cat image-recipe/build/auto-installer/archipelago-rootfs.recipe.sha256
```
**Caveat — do not compute this hash from the repo file.** `image-recipe/build-debian-iso.sh`
copies the archived builder to a temp path and rewrites its relative paths before exec'ing it,
and `RECIPE_HASH` hashes `"$0"` — the rewritten copy. The hashed region contains 35 such
rewritten path expressions, and `SCRIPT_DIR` is substituted with an absolute path, so the hash
is specific to the build host and checkout location. For reference, hashing the region of the
committed repo file directly gives `d2dc4df5427fe73d48227aab08cdf6debfe8dd554e6b18e3718f8d37ea9d675c`,
which is **expected to differ** from the stamp above.
---
## Result
_Paste the raw output of steps 2, 4, 5 and 6 here, then set the status at the top of this
document to VERIFIED with the date and build-host label._
```text
(pending — not yet run on a build host)
```
**Verdict:** _pending_
---
## What this does and does not prove
**Proves (once run):** the rootfs tar extracted verbatim onto every disk flashed from the ISO
carries no SSH host key, no TLS private key and no populated machine-id — so a first-boot
regeneration failure degrades to "no key, the service refuses to start" rather than
"fleet-shared key, silently", which is the substance of F-03.
**Does not prove:** that two nodes flashed from the same ISO actually end up with different
keys. That is audit item **C-3** (§779) and needs two physical machines; it remains
separately UNVERIFIED. C-4 is a build-host check only.
### Guidance for C-3: SSH and TLS are now equally sharp signals
An earlier revision of this document said SSH host keys were the sharper divergence signal for
C-3, because the installer had a per-install TLS fallback that would produce a differing cert
even if first-boot generation had failed. **That asymmetry no longer exists.**
There is now exactly one producer of each secret — `gen_tls()` and `gen_ssh()` inside
`first-boot-secrets.sh` — and no other code in the ISO build creates either. The Dockerfile no
longer bakes a TLS keypair and the installer's "ensure SSL cert exists" block is gone. So for
C-3, treat both the same way:
```bash
# on each node
ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -lf -
openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256
```
**Pass:** both fingerprints differ between the two nodes. **Fail:** either matches — a matching
TLS fingerprint is now exactly as damning as a matching host key, whereas before it could have
been explained away by the fallback.
Also check, on each node, that the run actually succeeded rather than merely being quiet:
```bash
ls -l /var/lib/archipelago/.secrets-regenerated # present on a healthy node
cat /var/lib/archipelago/first-boot-secrets.failed 2>&1 # absent on a healthy node
systemctl status archipelago-first-boot-secrets.timer # enabled; the self-heal path
```
The audit's original C-3 fail condition — a `WARNING:` line in the log alongside an existing
marker — can no longer occur by construction: the marker is only written when both generators
succeeded. If you ever see that combination, the fix has been reverted.
docs/security/KEY-03-SIGNING-POSTURE.md
-448
View File
@@ -1,448 +0,0 @@
# KEY-03 — Signing posture after the Bitcoin Core wallet deletion
> **What this document is.** The evidence-backed record of how Archipelago's Bitcoin signing
> posture stands after Phase 10 KEY-03. It supersedes, for the Bitcoin Core wallet specifically,
> the target state described in `docs/security/PSBT-SIGNING-ARCHITECTURE.md` §8 Phase 1 — that
> phase planned to *convert* Core's wallet to watch-only; **D-07b deleted the path instead.**
>
> **Governing decisions:** `.planning/phases/10-key-material-hardening/10-CONTEXT.md`
> **D-07b** (final KEY-03 scope — delete, do not migrate) and **D-07c** (the deferred BDK cold
> vault, recorded so it is not lost with the code). D-07b supersedes D-07 and D-07a's conditional
> migration.
>
> **Audit finding closed:** F-13 (High) —
> `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md:604`, remediation register R-04.
---
## Bitcoin Core wallet path — deleted (D-07b)
### What was deleted
| Symbol | Kind | Location before deletion |
|---|---|---|
| `handle_bitcoin_init_wallet_from_seed` | `async fn` | `core/archipelago/src/api/rpc/bitcoin.rs:161-295` |
| `"bitcoin.init-wallet-from-seed"` | JSON-RPC dispatch arm | `core/archipelago/src/api/rpc/dispatcher.rs:122-124` |
### The defect (F-13)
The handler loaded the encrypted seed, derived the **BIP-84 account extended private key**
(`crate::seed::derive_bitcoin_xprv`, `bitcoin.rs:188`), stringified it (`:189`), and imported
`wpkh(xprv/0/*)` and `wpkh(xprv/1/*)` (`:230-231`) into a Bitcoin Core descriptor wallet created
with `disable_private_keys = false` (`:203`) and an **empty** wallet passphrase (`:205`).
The result was a **second copy of the node's spending key**, persisted in Core's `wallet.dat`
inside the Bitcoin container's data volume, with no Argon2 passphrase — while the first copy sits
in the daemon's Argon2 + ChaCha20-Poly1305 envelope written `0600`
(`core/archipelago/src/seed.rs:238-269`, `:318-324`). That duplication, into weaker protection,
was the entire finding.
### Evidence that deletion was the right close (re-established for this task, not inherited)
The four D-07a evidence points, verified again against the tree before anything was removed:
**1. No caller anywhere.** Repo-wide search across `core/`, `neode-ui/src`, `scripts/`, `web/`,
`apps/`, `tests/` and `docs/`, excluding `core/target`, `node_modules` and `.git`:
```
$ grep -rn 'bitcoin\.init-wallet-from-seed' core/ neode-ui/src scripts/ web/ apps/ tests/ docs/
core/archipelago/src/api/rpc/dispatcher.rs:122: "bitcoin.init-wallet-from-seed" => {
$ grep -rn 'handle_bitcoin_init_wallet_from_seed' core/ neode-ui/src scripts/ web/ apps/ tests/ docs/
core/archipelago/src/api/rpc/bitcoin.rs:161: pub(super) async fn handle_bitcoin_init_wallet_from_seed(
core/archipelago/src/api/rpc/dispatcher.rs:123: self.handle_bitcoin_init_wallet_from_seed(params).await
docs/UNIFIED-TASK-TRACKER.md:208: §8 Phase 1). `handle_bitcoin_init_wallet_from_seed` passes
docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md:607:(`handle_bitcoin_init_wallet_from_seed`):
docs/security/PSBT-SIGNING-ARCHITECTURE.md:147: `handle_bitcoin_init_wallet_from_seed`, `core/archipelago/src/api/rpc/bitcoin.rs:161-294`).
```
Exactly one occurrence of the method name (its own dispatcher registration) and two of the symbol
in code (its definition and the dispatcher call). The three remaining symbol hits are prose in
documentation — the audit, the task tracker, and the PSBT architecture spec — not callers. No
frontend, script, test or other Rust module invoked it.
**2. LND is the wallet the product actually drives.** Across all of `neode-ui/src`, every
`bitcoin.*` RPC call is read-only status: `bitcoin.getinfo` (14 call sites),
`bitcoin.prune-status` (3), `bitcoin.onion` (1). There are **no** `bitcoin.*` wallet operations.
The wallet UI (`Web5Wallet.vue`, `SendBitcoinModal.vue`) sends via `lnd.sendcoins`, estimates via
`lnd.estimatefee`, and reads balance via `lnd.getinfo`.
**3. The wallet it creates never existed on the reference node.** Verified live on
**archi-dev-box, 2026-08-02**, against the running `bitcoin-knots` container (read-only RPCs
only — see the census section for the exact commands and the standing ban on
`listdescriptors true`):
```
listwalletdir → { "wallets": [ "gatewayd-02004b91…", "gatewayd-03443c0c…", "" ] }
listwallets → [ "" ]
```
**There is no wallet named `archipelago`** — the handler's default `wallet_name`
(`bitcoin.rs:170-173`). It has never run on this node. `getwalletinfo` on the one loaded wallet
(the unnamed default) reports:
```
walletname: "" blank: true keypoolsize: 0
txcount: 0 balance: 0.00000000
descriptors: true private_keys_enabled: true
```
`blank: true` with `keypoolsize: 0` and `txcount: 0` is Bitcoin Core's own statement that **no
key was ever imported into it and no transaction ever touched it**. The two `gatewayd-*` entries
are Fedimint gateway wallets, unrelated to the BIP-84 path. The `wallet.dat` at the datadir root
is Core's own legacy default-wallet location, not this handler's output.
**This is one node.** The same check was subsequently run across the reachable fleet — see the
census below: **4 nodes examined and clear, 6 unreachable and therefore unknown.**
**Supporting history evidence:** `git log -S "init-wallet-from-seed"` scoped to
`core/archipelago/src/api/rpc/dispatcher.rs` and `neode-ui/src` returns exactly one commit —
`19dcfd4f feat: BIP-39 master seed for unified key derivation`, the commit that **added** it. No
frontend wrapper was ever written: it was built and never wired up.
**4. It was never remotely reachable.** The endpoint is absent from `UNAUTHENTICATED_METHODS`
(`core/archipelago/src/api/rpc/middleware.rs:5-40`) — so it required an authenticated session —
**and** it additionally re-verified the user's password before touching the seed
(`self.auth_manager.verify_password(password)`, `bitcoin.rs:176-179`). **F-13 was therefore
key-at-rest duplication, not an exposed endpoint.** That is why it was rated High rather than
Critical, and why deleting it is a hardening measure rather than an incident response.
### What was *not* wrong with it
Worth stating so the record is fair, and so the next reader does not mistake the lesson. The
in-memory handling of the xprv string was **careful**: it was zeroized on the error path
(`bitcoin.rs:222`) and on the success path (`:284`), matching the standard set elsewhere in
`seed.rs`. The wallet type was also correct — `createwallet` already passed `descriptors = true`
(`:207`), which is the right foundation.
**The defect was which key went into the wallet, not how the key was held in memory or what kind
of wallet it was.** A watch-only rewrite (xpub + `[fingerprint/derivation]` key origin) would
have been a legitimate fix. Deletion was chosen over rewrite because the endpoint had no caller,
no consumer, and no product role: rewriting it would have produced a correct implementation of
something nothing uses, and left a wallet-creating code path to be maintained and re-audited
forever.
### How F-13 is closed
**By removal, not by conversion to watch-only.** After this change there is no code path in the
daemon that writes the BIP-84 account private key into Bitcoin Core. The only on-node copy of
that key is the daemon's Argon2 + ChaCha20-Poly1305 envelope.
**No migration was performed and none is planned.** D-07's parity-proof migration and its
one-way checkpoint are **withdrawn** (D-07b) — there is no wallet to migrate. If a fleet node is
ever found holding a descriptor wallet this handler created, that is a **finding to surface and
stop on**, not a trigger to auto-migrate: it would mean the endpoint was invoked by hand and that
node's spending key is duplicated in Core, which deserves a human decision rather than an
automated rewrite of a wallet that may hold funds.
### This deletion removes code, not wallets
Stated explicitly so nobody reading the change later has to wonder whether it was destructive:
> **Nothing on disk is touched.** No `wallet.dat` is modified, unloaded or removed. No funds
> move. No LND state, secret, descriptor or seed is altered. The change removes a Rust function
> and a `match` arm — the *path* by which a private key could be imported into Bitcoin Core —
> and nothing else.
This holds even on a hypothetical node where the endpoint had been invoked by hand: deleting the
handler destroys nothing there either. It closes the door; it does not clean the room. Cleaning
up such a wallet, if one is ever found, is a separate human decision (see the census below), and
CLAUDE.md's **"migrations never destroy data"** invariant is not engaged by this change because
there is no migration.
### What deletion does to D-08 and D-09
Neither decision lapses; both are satisfied by a different mechanism.
- **D-08** asked that the spending key exist in exactly one place, with an opt-in air-gapped
path. Deleting the Core import achieves the first half outright. The opt-in path is LND's
existing PSBT round trip, not a Core watch-only wallet — see the next section, including the
recorded verdict on how far that actually goes today.
- **D-09** required a `[fingerprint/derivation]` key origin on emitted descriptors so a hardware
signer can locate its key. With Core's descriptors deleted there are **no Archipelago-emitted
descriptors left to annotate**, so D-09's actual protection moves to the PSBT itself. That is
why `lnd.create-psbt` now inspects and reports the key-origin data its PSBT carries
(`psbt_key_origin_report`, `core/archipelago/src/api/rpc/lnd/wallet.rs`).
### `derive_bitcoin_xprv` is retained deliberately (D-07c)
`crate::seed::derive_bitcoin_xprv` (`core/archipelago/src/seed.rs:231`) lost its only non-test
caller and was **kept**, marked `#[allow(dead_code)]` with the reason in its doc comment. It is
covered by existing tests (`seed.rs:601-602`, `:856`) and it is the derivation **D-07c's deferred
BDK cold vault** — a descriptor wallet in the daemon using the node's own ElectrumX app
(`apps/electrumx`, `electrs_status.rs`) as chain source — will need.
D-07c was considered and deliberately deferred out of Phase 10 (it needs its own phase: a new
dependency and a new UI surface). It is recorded here, and in the function's doc comment, so the
option is not quietly lost along with the code that was deleted. The alternative shape — LND
watch-only via `importaccount` plus remote signing — was considered and rejected for coupling
cold storage to LND's upgrade path.
---
## LND PSBT round trip — what is covered
With Core's wallet deleted, LND is the only wallet Archipelago has, and its PSBT round trip is
the only external-signer path that exists. This section records what that path actually consists
of, what is tested, and — the question that decides whether any of it is an air gap — whether an
externally-held signer can sign a default node's PSBT at all.
### Per-step coverage map
Round trip: **fund → export → sign offline → import → finalize → broadcast.**
| # | Step | Where it lives | `file:line` | Automated test coverage |
|---|---|---|---|---|
| 1 | **Fund** — build a funded PSBT via LND WalletKit `/v2/wallet/psbt/fund` | `lnd.create-psbt` handler | `core/archipelago/src/api/rpc/lnd/wallet.rs:605`; dispatch arm `api/rpc/dispatcher.rs:136` | **Untested.** No LND mock exists; the handler's request/response handling is exercised only by hand. |
| 1a | **Inspect** — report BIP-32 key origin on the funded PSBT | `psbt_key_origin_report` + wiring | `lnd/wallet.rs:1186` (fn), `:1169` (struct), `:705` (call site), `:737` (response field) | **Tested.** 3 unit tests, below. |
| 2 | **Export** — hand the base64 PSBT to the user | UI renders `psbt_base64` for copy | `neode-ui/src/api/rpc-client.ts:407-423`; `neode-ui/src/views/web5/Web5SendReceiveModals.vue:308` | **Partial.** `neode-ui/src/api/__tests__/rpc-client.test.ts:319-323` asserts only that the client calls the method `lnd.create-psbt`; it does not test the payload or the rendering. |
| 3 | **Sign offline** — external signer produces a signed PSBT | **Not in this repo.** No first-party signer ships today. | — | N/A |
| 4 | **Import** — user pastes the signed PSBT back | textarea → `signedPsbtInput` | `Web5SendReceiveModals.vue:102`, `:419-424` | **Untested.** |
| 5 | **Finalize** — `/v2/wallet/psbt/finalize` | `lnd.finalize-psbt` handler | `lnd/wallet.rs:743`; dispatch arm `dispatcher.rs:137` | **Untested.** |
| 6 | **Broadcast** — `/v2/wallet/tx`, in the same handler | `handle_lnd_finalize_psbt` tail | `lnd/wallet.rs:795` | **Untested.** |
| — | **Rate limiting** — both endpoints at 5 calls / 300s | `RateLimiter` defaults | `core/archipelago/src/rate_limit.rs:68-69` | **Untested for these two methods specifically.** |
**Stated plainly, because an untested path must not be described as verified:** of the six steps,
**one** (the key-origin inspection added by this plan) has automated coverage in the Rust
crate. Steps 1, 4, 5 and 6 have **none** — no test exercises the LND REST calls, the finalize
handler, or the broadcast. Step 2's only test asserts a method name. **No end-to-end test of the
round trip exists**, and none of it has been verified against a real hardware signer.
There is also **no air-gap transport**: no animated QR encode/decode, no `.psbt` file
download/upload. Export and import are copy-paste of base64 in a textarea. The BC-UR v2 / BBQr
design in `PSBT-SIGNING-ARCHITECTURE.md` §4 is unimplemented.
### New tests added by this plan
In `core/archipelago/src/api/rpc/lnd/wallet.rs`'s `mod tests`, with fixtures built
programmatically from the `bitcoin` crate rather than pasted as opaque base64:
| Test | Asserts |
|---|---|
| `psbt_without_derivations_reports_no_key_origin` | A one-input unsigned PSBT with no `bip32_derivation` reports `inputs_with_key_origin: 0` and `all_inputs_have_key_origin: false`. |
| `psbt_with_derivations_reports_key_origin` | The same PSBT with a `(Fingerprint, DerivationPath)` inserted on input 0 reports `1/1` and `true`. |
| `malformed_psbt_is_an_error_not_a_panic` | Non-base64, truncated-PSBT and empty inputs all return `Err`, never panic. |
```
running 3 tests
test api::rpc::lnd::wallet::tests::psbt_with_derivations_reports_key_origin ... ok
test api::rpc::lnd::wallet::tests::psbt_without_derivations_reports_no_key_origin ... ok
test api::rpc::lnd::wallet::tests::malformed_psbt_is_an_error_not_a_panic ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 1014 filtered out
```
`lnd.create-psbt` now returns an additive `key_origin` field:
```json
"key_origin": { "input_count": 1, "inputs_with_key_origin": 0, "all_inputs_have_key_origin": false }
```
It is computed **best-effort**: a decode failure degrades to `null` and logs a warning, never to
an error — a user's send must not fail because an inspection helper could not parse something.
When `all_inputs_have_key_origin` is false the handler emits a `tracing::warn!` with the counts,
because that is the exact condition under which a hardware signer refuses the PSBT. Existing
response fields are unchanged; `handle_lnd_finalize_psbt` and `handle_lnd_create_raw_tx` (the
sibling that deliberately auto-signs with LND's hot keys) were not touched.
### Can an external signer actually sign a default node's PSBT? — **No, not today**
This is the question that separates "we have PSBT plumbing" from "we have air-gapped custody",
and the two must not be allowed to blur.
**Verdict: on a default Archipelago node, an externally-held signer cannot meaningfully sign a
PSBT produced by `lnd.create-psbt`.** The evidence:
1. **The PSBT is funded from LND's own wallet.** `lnd.create-psbt` POSTs to LND's WalletKit
`/v2/wallet/psbt/fund` (`lnd/wallet.rs:672`), which selects UTXOs belonging to **LND's**
wallet. The keys for those inputs are the keys LND holds.
2. **LND's wallet on every node is a full key-holding wallet, created locally.**
`container::lnd::ensure_wallet_initialized` (`core/archipelago/src/container/lnd.rs:86`) calls
`init_wallet_via_rest`, which POSTs `/v1/initwallet` with a `cipher_seed_mnemonic`
(`container/lnd.rs:504-516`) and persists the aezeed backup (`:523-525`). That is a normal
wallet with private keys, not a watch-only one.
3. **No node's `lnd.conf` carries a remote-signing block.** The config Archipelago generates
(`container/lnd.rs:64-79`) contains `bitcoin.node=bitcoind` and the bitcoind RPC settings, and
**no `remotesigner.*` keys at all**.
4. **Nothing in the repo provisions watch-only LND.** A search of `apps/`, `scripts/`,
`core/archipelago/src` and `image-recipe/` for `remotesigner`, `createwatchonly` and
`nochainbackend` returns **zero matches**. There is no code path, script or manifest that sets
any node up this way.
An external signer could only sign these inputs if LND were first provisioned **watch-only
against that signer** — `remotesigner.*` on the node plus `lncli createwatchonly` from the
signer's exported accounts, with the level-3 accounts and the p2tr import step described in
`PSBT-SIGNING-ARCHITECTURE.md` §5.1-5.2. **No fleet node is so provisioned.**
**What therefore ships today is the PSBT *transport*, not air-gapped custody.** The round trip is
real and rate-limited, and it is genuinely useful for signing a PSBT whose inputs belong to some
*other* wallet — but on a default node the signer that holds the input keys is LND itself, so
routing the PSBT out to an external device and back adds a step without moving custody anywhere.
The gap between here and D-08's opt-in air-gapped path is **provisioning, not plumbing**, and
that provisioning is out of scope for Phase 10 (it is `PSBT-SIGNING-ARCHITECTURE.md` §8 Phase 6).
Nothing in the UI currently claims otherwise, and nothing added by this plan does either. If
copy is ever written for this flow, it must not describe it as cold storage on the strength of
the PSBT round trip alone.
### Lightning channel, revocation and HTLC keys are not air-gappable — at all
This is a standing constraint, not a caveat, and it survives every change in this document.
> **A Lightning node's channel, revocation and HTLC keys must sign in real time to answer
> counterparty commitments. They cannot be air-gapped.** A routing node cannot tolerate a
> human-in-the-loop signing step: a delayed response to a commitment update risks a force-close,
> and a missing revocation risks loss. LND remote signing **relocates** these keys to a hardened
> host — it does **not** cool them. There is no configuration, present or future, in which a
> live Lightning node's channel keys are cold.
This is the same limit stated in `PSBT-SIGNING-ARCHITECTURE.md` §5.1 ("Air-gap channel /
revocation / HTLC keys — **No**") and §5.4, whose honesty table remains correct and unmodified.
The consequence for user-facing copy, quoted from §5.4 and repeated here so it cannot be lost:
> *A Lightning routing node's channel keys are necessarily hot. Remote signing moves them to a
> hardened machine; it does not make them cold. Only your on-chain balance can be genuinely
> protected by an offline signer.*
**No wording in this document, or in any document this phase touches, may imply that Lightning
funds can be held cold.** A user who believes their Lightning balance is cold will keep more in
it than they otherwise would, which is exactly the miscalibration that turns an incident into a
loss.
---
## Fleet census — Core descriptor wallets
**Status: run 2026-08-02 — 4 nodes examined and CLEAR, 6 nodes UNCHECKED. No escalation.**
This section answers one question per node: *does this node hold a Bitcoin Core descriptor wallet
that the deleted wallet-init handler created, and does it hold private keys?* It is recorded per
node rather than assumed, because deletion closes the door but does not tell us whether anyone
walked through it before.
The nodes that could **not** be examined are listed with their reasons, not omitted. A census
that quietly drops its failures is worthless — an auditor must be able to see exactly which
machines were looked at and which were not.
### Hard constraint on every command in this census
> **Never run `listdescriptors true`.** The `true` argument makes Bitcoin Core return the
> descriptors **including private keys**, which would print an xprv to a terminal and into a
> transcript — creating the exact exposure this census exists to measure.
> `listwalletdir`, `listwallets`, `getwalletinfo` and `listdescriptors` **with no second
> argument** answer the question completely.
>
> If any output unexpectedly contains a string beginning `xprv`, **stop immediately, do not
> paste it**, and report only that it occurred.
### Commands (re-runnable by an auditor)
Per node, against the Bitcoin Core / Knots container:
```bash
# 0. Does the handler's wallets directory exist at all? An absent directory is
# itself a complete answer for that node — paste the output as-is.
ls -la /var/lib/archipelago/bitcoin/wallets/ 2>&1
# bitcoin-cli is NOT on $PATH inside the container. On archi-dev-box (Knots
# 29.3) it lives at:
# /opt/bitcoin-29.3.knots20260210/bin/bitcoin-cli
# The RPC user is `archipelago`; the password is read from
# /var/lib/archipelago/secrets/bitcoin-rpc-password
# — reference that path, never the value, and prefer -stdinrpcpass so the
# password never appears in a process list or shell history.
# 1. Every wallet on disk, loaded or not.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass listwalletdir
# 2. Currently loaded wallets.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass listwallets
# 3. Per wallet returned: record walletname, private_keys_enabled, descriptors,
# blank, keypoolsize, txcount, balance.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass -rpcwallet=<name> getwalletinfo
# 4. ONLY for a wallet with private_keys_enabled: true — NOTE: no second argument.
# Record descriptor prefixes (`wpkh(...`) only, never a full key string.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass -rpcwallet=<name> listdescriptors
# 5. Which Bitcoin app and version.
bitcoin-cli -rpcuser=archipelago -stdinrpcpass getnetworkinfo | head
```
### Results — examined, 2026-08-02 (4 nodes, all CLEAR)
Run by the operator over Tailscale, read-only RPCs only.
| Node | Tailscale IP | Container | `listwalletdir` | `listwallets` | `archipelago` wallet? | Default wallet state | Verdict |
|---|---|---|---|---|---|---|---|
| **archi-dev-box** | `100.69.68.39` | `bitcoin-knots` | 2× `gatewayd-*`, `""` | `[ "" ]` | **No** | `blank: true`, `keypoolsize: 0`, `txcount: 0`, `balance: 0.00000000`, `descriptors: true` | **CLEAR** |
| **shorty-s** (`.228`) | `100.64.204.114` | `bitcoin-knots` | 1× `gatewayd-*`, `""` | `[ "" ]` | **No** | same | **CLEAR** |
| **archy-x250-beta** | `100.72.136.5` | `bitcoin-core` | 1× `gatewayd-*`, `""` | `[ "" ]` | **No** | same | **CLEAR** |
| **archy-x250-pa** | `100.89.209.89` | `bitcoin-core` | 1× `gatewayd-*`, `""` | `[ "" ]` | **No** | same | **CLEAR** |
On every examined node there is **no wallet named `archipelago`** — the deleted handler's default
`wallet_name`. The only named wallets are Fedimint `gatewayd-*`, unrelated to the BIP-84 path.
The one loaded wallet on each node is Core's unnamed default. It does report
`private_keys_enabled: true`, but also `blank: true` with `keypoolsize: 0`, `txcount: 0` and
`balance: 0.00000000` — **Bitcoin Core's own statement that no key was ever imported into it and
no transaction ever touched it.** It is not the deleted handler's output, and it holds nothing.
**The result holds across two container vintages** — `bitcoin-knots` on two nodes and
`bitcoin-core` on two others. That matters: it is not four copies of one image behaving
identically, so the finding is a property of the fleet rather than an artefact of a single build.
**No key material appeared in any output, and `listdescriptors true` was never run.**
### Not examined, 2026-08-02 (6 nodes, with reasons)
| Node | Tailscale IP | Why not checked |
|---|---|---|
| framework-pt | `100.65.115.109` | `Permission denied (publickey,password)` — SSH password rotated, not held |
| archipelago-1 | `100.82.34.38` | `Permission denied (publickey,password)` |
| archipelago | `100.70.96.88` | `Permission denied (publickey,password)` |
| archy-dev-pa | `100.64.83.15` | `Permission denied (publickey,password)` |
| archipelago-5 | `100.114.134.21` | Timed out during SSH banner exchange |
| archy-x250-dev | `100.113.100.55` | Offline — Tailscale reports last seen 2 days prior |
**Password authentication was deliberately not attempted on any of these.** Several fleet nodes
lock PAM quickly on a wrong password, and locking an in-use production node out is a worse
outcome than an incomplete census. These are recorded as UNCHECKED, **not** as clear.
### Conclusion, at the strength the evidence supports
> **No examined node holds a wallet created by the deleted handler, and no examined node holds
> any wallet with keys or funds.** Four nodes, across two container vintages, on 2026-08-02.
**This is deliberately not a claim that "the fleet is clear."** Six nodes were not examined, and
an unexamined node is unknown, not safe. F-13 is closed **by deletion** — the code that could
create such a wallet is gone from every future build, which is true regardless of the census —
and the census adds that no such wallet was found where anyone could look.
### Standing item — finish the census
The six unchecked nodes remain open. **Homed in `docs/UNIFIED-TASK-TRACKER.md`** (the project's
canonical "what's open" list) as *"Finish the Core-wallet fleet census — 6 nodes unchecked"*,
rather than only here, so it is visible to someone who is not already reading a security
document. It is flagged there as a natural fold-in for **KEY-04's on-node work**, which needs
node access anyway — but it is tracked independently so it does not vanish if KEY-04 is
re-scoped.
Re-run the read-only procedure above when credentials or connectivity allow.
### Standing rule if a wallet is found
If any node reports a wallet named `archipelago` (or any descriptor wallet with
`private_keys_enabled: true` that this handler plausibly created), that is a **finding**:
1. **Stop.** Record it here with the node label and wallet name.
2. **Raise it as a blocker.** KEY-03 does not close until a human decides what to do about it.
3. **Do not migrate, unload, rescan or modify it.** D-07b withdrew the migration deliberately.
Rewriting a wallet that might hold funds is exactly the kind of decision that belongs to a
human, and CLAUDE.md's "migrations never destroy data" invariant applies the moment anyone
touches it.
Such a wallet would mean the endpoint was invoked manually before this plan deleted it, and that
node's spending key is duplicated in Core outside the Argon2 envelope.
docs/security/KEY-05-ENTROPY-ENFORCEMENT.md
+12 -7
View File
@@ -1,13 +1,18 @@
# KEY-05 — Entropy enforcement: per-site classification and mechanism record
**Requirement:** ROADMAP `KEY-05`. **Plan:** `.planning/phases/10-key-material-hardening/10-06-PLAN.md`.
**Requirement:** ROADMAP `KEY-05`.
**Supersedes:** backlog `R-13`. **Absorbs:** `R-05` (duplicate-`rand` visibility) and `R-09`
(CSPRNG-readiness record). **Resolves:** `F-10a` in
`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`, which recorded raw match counts and
**deliberately declined to classify them**.
(CSPRNG-readiness record). **Resolves:** `F-10a` from the internal entropy and
seed-generation audit, which recorded raw match counts and **deliberately declined
to classify them**.
**Tree state this document was derived against:** `HEAD = c5a82cba` (2026-08-02).
**Update:** every `migrate` disposition in the table below has since been applied.
No `rand::random()` / `rand::thread_rng()` call remains in production `archipelago`
code — each draws through `entropy::draw_key_bytes` from a named `OsRng`, and
`core/clippy.toml` now bans both APIs, so a regression fails the build.
---
## Nothing here is broken today
@@ -44,7 +49,7 @@ they are enabled wrongly. Both are gated behind Task 5, a `gate="blocking-human"
## Source precedence
`.planning/phases/10-key-material-hardening/10-CONTEXT.md` (2026-08-01) lists **F-07 / R-05**
The Phase 10 hardening work lists **F-07 / R-05**
and **F-10 / R-13** under `## Deferred Ideas`. KEY-05 was added to the ROADMAP on
**2026-08-02**, after that context was gathered, and explicitly absorbs R-05 and supersedes
R-13. The ROADMAP requirement is the later and governing artifact.
@@ -206,8 +211,8 @@ lines are `core/archipelago/src/mesh/x3dh.rs:99` and `:113` —
produced there; `:100` and `:114` draw only the `u32` `id` fields of `SignedPrekey` and
`OneTimePrekey`. They remain in scope — they are values that go on the wire — but the
characterisation "X3DH key agreement — key material" overstates these two specific lines.
(The audit has since been corrected in place at `ENTROPY-SEED-AUDIT-2026-07-31.md:508`; this
section records the derivation independently.)
(The internal audit has since been corrected; this section records the derivation
independently.)
---
docs/security/LND-MACAROON-ROTATION.md
+166
View File
@@ -0,0 +1,166 @@
# Rotating this node's Lightning credentials
A Lightning macaroon is a **bearer token**: whoever holds one can spend from the
node's wallet. There is no revocation list and no expiry. If a macaroon is ever
read by something you do not control — a leaked endpoint, a screenshot, a phone
that has since been lost, an app that ran a version with a published
vulnerability — that ability persists until the macaroons are rotated.
Rotation is therefore a **routine operator action**, not an emergency procedure.
Two paths do the same work:
| Path | Use when |
|---|---|
| **Dashboard** — Settings → *Lightning credentials* | Normal case. Password-confirmed, shows progress, repairs BTCPay for you. |
| **`scripts/security/rotate-lnd-macaroon.sh`** | No dashboard reachable, or you want a detect-only report. |
## What rotation actually does
LND derives every macaroon it issues from a root key in `macaroons.db`. Remove
that root key plus the issued `*.macaroon` files, restart, and LND mints a fresh
root key and a fresh set of macaroons when the wallet unlocks. Every macaroon
issued before that moment — including any an attacker holds — stops verifying.
## Why your funds and channels survive
Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
state in `channel.db`; channels are secured by the node's identity and channel
keys, none of which are derived from the macaroon root key. Neither database is
opened, moved or deleted.
Both paths **prove** this rather than asserting it: they record the node's
identity pubkey and its channel census before rotating, and refuse to report
success if either differs afterwards.
Two details in that check are deliberate and should not be "tightened":
- **Channels are compared as a total, not as `num_active_channels`.** The active
count only counts channels whose peer is currently online, so it legitimately
dips for minutes after *any* restart while peers reconnect. Asserting on it
alone would abort a perfectly healthy rotation.
- **`wallet.db` is not compared byte-for-byte.** btcwallet records chain-sync
progress inside it, so the file changes on every start. Asserting byte-identity
would fire a frightening false alarm on a completely healthy rotation.
## What it never does
- No macaroon **content** reaches a response, an error, a log line, or the
progress feed the dashboard polls. Everything reported is a SHA-256 digest or a
byte count — enough to prove the material changed without disclosing it to
whoever is reading the screen.
- No path from "rotate my credentials" to "delete my wallet". LND's boot path
self-heals a wallet no candidate password can open by wiping and recreating it;
correct for an unattended boot, catastrophic here. Rotation unlocks through
`container::lnd::unlock_existing_wallet_no_wipe`, so a wallet whose password
this node does not hold surfaces as a **failed rotation** with the wallet
intact.
## The BTCPay coupling — the part that bites
**BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot
self-heal. LND's data directory is owned by its container's mapped uid, so BTCPay
cannot bind-mount the macaroon file (EACCES across the userns boundary). The
connection string therefore carries the macaroon as hex:
```
type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<hex>
```
delivered as the `btcpay-lnd-connection` secret file. Rotate the macaroons and
that copy becomes a dead credential. Nothing notices on its own, because the
daemon only regenerates this secret when LND's **TLS cert thumbprint** changes —
and macaroon rotation does not touch the cert.
The resulting state is the dangerous one: **BTCPay is up, LND is up, both report
healthy, and every Lightning invoice BTCPay tries to create fails.**
Repair needs two things, and one without the other is cosmetic:
1. **Rewrite the secret** (`container::lnd::rewrite_btcpay_lnd_connection_secret`).
This is what makes the change visible: `secret_env_hash` is derived from the
resolved secret contents, so a changed file reads as label drift on the
running container.
2. **Recreate the container.** `btcpay-server` is on the restart-sensitive list,
and the reconcile loop runs in `ExistingOnly` mode *always* — boot and
periodic alike — where env drift on a restart-sensitive app is detected and
then deliberately skipped. Rewriting the secret alone therefore changes
nothing that is running. Observed directly on a development node, once per
tick, for half an hour:
```
container drift detected during boot reconcile; leaving running
restart-sensitive app untouched app_id=btcpay-server
```
The dashboard path calls
`ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
the flag the drift check consults to override restart-sensitivity. It is the
same carve-out FED-07 added for the Fedimint gateway, and the reasoning is
identical: restart sensitivity protects apps that are *working*, and this one
is working only in appearance.
**The shell script cannot set that in-process flag**, so it does the equivalent
from outside: it deletes the secret (the daemon regenerates it within a tick),
then removes the `btcpay-server` container so the orchestrator's own
desired-state recovery rebuilds it around unchanged data. That recovery is what
makes this safe rather than a hand-rolled remove-and-run — it fires because the
app is still installed and was in the last running-containers snapshot. The
script then prints the commands to confirm it actually happened, because a
failure here is invisible.
## Slow nodes: the unlock budget
LND opens `channel.db`, `graph.db` and `wallet.db` before it serves the unlocker
at all, and on a busy node that is genuinely slow — **2m38s measured on a box
running 30 containers**. The unlock helper used to give up after ~60s, which on
such a node could never succeed.
That timeout was not a harmless retry. Reconcile records the post-start hook as
failed, restarts LND, and the slow database open starts over: a restart loop that
leaves the wallet permanently locked and every LND-dependent app (BTCPay's
internal node included) broken, on exactly the nodes least able to afford it.
The not-ready budget is now ~10 minutes (`UNLOCK_NOT_READY_ATTEMPTS`). Waiting
longer costs nothing, because a genuinely wrong password still exits on the first
pass through the candidate list — the `all_rejected` fast path is untouched.
## Verifying a rotation
The dashboard shows all of this. From a shell:
```bash
# 1. Fingerprint changed (digest only — never print the macaroon)
sudo sha256sum /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon
# 2. Same node, same channels
podman exec lnd lncli --network=mainnet getinfo \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["identity_pubkey"], \
d["num_active_channels"] + d["num_inactive_channels"], d["num_pending_channels"])'
# 3. BTCPay is carrying the CURRENT macaroon, not the rotated-out one
CUR=$(sudo od -An -v -tx1 /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon | tr -d ' \n')
SEC=$(sudo sed -n 's/.*macaroon=\([0-9a-f]*\).*/\1/p' /var/lib/archipelago/secrets/btcpay-lnd-connection)
[ "$CUR" = "$SEC" ] && echo "current" || echo "STALE — BTCPay's Lightning is broken"
# 4. BTCPay was actually recreated (a silent failure looks like success)
podman inspect btcpay-server --format '{{.Created}}'
```
Check 3 is the one people skip, and it is the one that fails.
## Afterwards
- **Re-pair every wallet app**, Zeus most importantly. Open the Lightning app in
the dashboard and scan the pairing QR again; it serves the new macaroon.
- **Delete the backup once re-pairing is done.** Both paths back the old material
up to `/var/lib/archipelago/lnd/macaroon-rotation-<stamp>` (0700) so a mistake
is recoverable. That directory holds the **old root key** and is still
sensitive: `sudo rm -rf <path>`.
## Related
- `docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md` — the leak that first made
rotation necessary, and the operator decision not to rotate the fleet for it.
- `scripts/security/rotate-lnd-macaroon.sh` — the shell path, including its
ordering guard (it refuses to rotate on a binary that still leaks
`/lnd-connect-info`, since the new macaroon would leak within seconds).
docs/security/PHASE-10-VERIFICATION-GUIDE.md
-267
View File
@@ -1,267 +0,0 @@
# Phase 10 — Independent Verification Guide
**Audience:** third-party security auditors, and the Archipelago team.
**Purpose:** verify the Phase 10 security claims *independently*, without trusting the
project's own test harness.
**Status:** LIVING — sections are marked ✅ verifiable now, ⏳ pending a plan still in
execution, or 🔒 hardware-gated. Do not read an unmarked absence as a passing result.
---
## 0. How to use this document
Every claim below follows the same four-part structure, and **all four parts matter**:
| Part | Why it exists |
|---|---|
| **Claim** | Stated so it can be falsified. A claim you cannot disprove is not a security claim. |
| **Reproduce the defect** | Check out the parent commit and demonstrate the bug. *A test that passes on both the fixed and unfixed code proves nothing.* |
| **Verify the fix** | Command + expected output, runnable without our harness wherever possible. |
| **Negative control** | Break the fix deliberately; confirm the check goes red on **exactly** that and nothing else. This is what separates verification from demonstration. |
**Do not skip "Reproduce the defect".** It is the only step that proves the fix addresses
something real, and it is the step most often omitted in security theatre.
### Trust posture
Where a claim can be checked from *outside* the codebase — an HTTP request from another host,
a `tar` listing, a file comparison across two machines — **prefer that over running our tests.**
Our tests are offered as convenience and as evidence of intent, not as proof. Every claim below
that can be externally checked says so explicitly.
---
## 1. Scope
### In scope — what Phase 10 claims
| ID | Claim | Severity | Status |
|---|---|---|---|
| KEY-01 | An already-provisioned node refuses every unauthenticated RPC that can mutate identity or credentials | **Critical** | ⏳ `10-01` in execution |
| KEY-02 | First-boot per-device secret generation is fail-closed, retried, self-healing, and has exactly one producer; the shipped rootfs contains no fleet-shared identity material | **High** | ✅ partially landed (`21043096`, `408b328c`), ⏳ single-producer + self-heal in progress |
| KEY-03 | The BIP-84 account private key is never imported into Bitcoin Core; the dead import path is deleted | **High** | ⏳ `10-05` in execution |
| KEY-04 | On-node evidence for C-3 / C-4 / C-6 | — | 🔒 hardware-gated |
| KEY-05 | A defaulted RNG cannot be inherited anywhere in the crate | Medium | ⏳ `10-06` not started |
### Explicitly NOT claimed
State these plainly so an auditor is not left inferring them:
- **Lightning custody is not air-gappable.** Channel, revocation and HTLC keys must sign in real
time to answer counterparty commitments. LND remote signing *relocates* those keys; it does not
make them cold. Any document implying otherwise is wrong.
- **No claim against a compromised kernel CSPRNG**, a malicious dependency in the supply chain,
memory disclosure on a running node, or physical access.
- **KEY-05 fixes a structural risk, not a live vulnerability.** `rand::random()`/`thread_rng()`
are ChaCha12 seeded from `getrandom(2)`; nothing in that finding is exploitable today. The
mitigation targets *future silent rebinding* of the entropy source.
- **Findings F-04 through F-12 are out of scope** for this phase and remain open. See
`ENTROPY-SEED-AUDIT-2026-07-31.md` remediation register (R-05..R-15) and
`docs/UNIFIED-TASK-TRACKER.md`.
---
## 2. Provenance
```bash
# The audit that motivated this phase
docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md # 103 file:line references
# The entropy fix that preceded the phase
git show 8b51b7e2 # seed.rs — explicit OsRng at the call site
# Phase 10 plans and locked decisions
.planning/phases/10-key-material-hardening/
```
`.planning/` is committed deliberately: an auditor can read *why* each decision was made,
including the ones that were reversed. `10-CONTEXT.md` records D-01..D-11 plus three
in-flight corrections (D-03a, D-07a/b/c) where our own earlier reasoning was wrong.
---
## 3. Tier 0 — verifiable on any checkout, no node required ✅
No hardware, no deploy. Start here.
### 3.1 First-boot secrets are fail-closed (KEY-02)
**Claim.** If per-device secret generation fails, the completion marker is **not** written and
the boot does not proceed as if it had succeeded.
**Reproduce the defect:**
```bash
git log --oneline -1 21043096 # the fix commit
git show 21043096^:image-recipe/_archived/build-auto-installer-iso.sh > /tmp/pre-fix.sh
grep -n 'touch .*MARKER' /tmp/pre-fix.sh
# Observe: the marker write is NOT inside the success branch — it runs regardless of outcome.
```
**Verify the fix:**
```bash
bash tests/first-boot-secrets/run-tests.sh
# Expect: passed: 3 failed: 0 (more cases once the self-heal work lands)
```
The harness extracts the heredoc body **from the builder itself**, so it exercises the bytes
that ship rather than a copy. Confirm that for yourself:
```bash
grep -n 'extracted .* lines from the builder' tests/first-boot-secrets/run-tests.sh
```
**Negative control:**
```bash
# Move `touch "$MARKER"` outside the success branch in the builder, then:
bash tests/first-boot-secrets/run-tests.sh
# Expect: FAIL: openssl fails every attempt -> MARKER-SET-ON-FAILURE
# passed: 2 failed: 1 EXIT=1
git checkout image-recipe/_archived/build-auto-installer-iso.sh
```
It must fail on **that case only**. A negative control that reddens everything is measuring
nothing.
### 3.2 Master-seed entropy is explicit (F-02, shipped)
**Claim.** Mnemonic generation draws from an explicitly-passed `OsRng`, not a
transitive-dependency default, and a test proves the injected RNG is the one consumed.
```bash
git show 8b51b7e2 -- core/archipelago/src/seed.rs # ~6 lines of production change
cd core && cargo test -p archipelago seed:: # expect 25 passed; 0 failed
```
**Reproduce the defect:** on `8b51b7e2^`, `MasterSeed::generate` calls
`bip39::Mnemonic::generate(24)`, which resolves to `&mut rand::thread_rng()` *inside* the bip39
crate — there is no seam to inject through, so the proving test cannot be written at all.
**Note for auditors:** the test module implements `rand::CryptoRng` for a counter RNG. That is a
deliberately false marker-trait promise, confined to `#[cfg(test)]` (`seed.rs:502`). KEY-05
retires it. Confirm containment:
```bash
grep -n 'CountingRng' core/archipelago/src/seed.rs # all hits must be after the cfg(test) at :502
```
### 3.3 Unauthenticated method inventory (KEY-01 context)
Read the authoritative list rather than trusting prose:
```bash
sed -n '/UNAUTHENTICATED_METHODS/,/];/p' core/archipelago/src/api/rpc/middleware.rs
```
Every entry is reachable without a session, RBAC check, or CSRF token. KEY-01's claim is that
those which can mutate identity or credentials refuse once the node is provisioned.
---
## 4. Tier 1 — requires a running node ⏳
Pending `10-01` and `10-02`. `10-02` produces `scripts/security/rpc-exposure-probe.sh` and
`docs/security/KEY-01-ON-NODE-VERIFICATION.md`.
**The external check that matters most (C-6).** From a *different host* on the same network,
against a node that has completed onboarding:
```bash
curl -sS -X POST http://<node>/rpc \
-H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"seed.restore","params":{"words":["<24 words>"]}}'
```
- **Before the fix:** the node accepts attacker-supplied words and overwrites `node_key`,
`nostr_secret` and the FIPS mesh key. This is the Critical finding.
- **After the fix:** refused, and the node's identity is byte-identical afterwards.
Verify byte-identity yourself rather than trusting a log line:
```bash
sha256sum /var/lib/archipelago/identity/node_key /var/lib/archipelago/identity/nostr_secret
# run before and after the request; the hashes must be unchanged
```
> ⚠️ **Do not run the "before" case against a node you care about.** It really does overwrite the
> identity. Use a disposable node — see `.planning/todos/pending/2026-08-01-archi-dev-box-as-fresh-test-node-without-iso.md`
> for standing up an isolated instance without flashing an ISO.
**Do not probe with `seed.status`.** The original audit's C-6 command used it; `seed.status` is
**not** in `UNAUTHENTICATED_METHODS`, so it returns 401 by design and would report the surface
closed while the real door stands open. Probe with a method that is genuinely on the
unauthenticated list.
**Non-regression, equally important:** a *fresh, un-onboarded* node must still complete
onboarding. The gate distinguishes provisioned from fresh; a fix that refuses on a fresh node
bricks first boot fleet-wide.
---
## 5. Tier 2 — ISO build host 🔒
Full procedure: `docs/security/KEY-02-ROOTFS-EVIDENCE.md` (C-4).
```bash
UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild
# then follow steps 2/4/5/6 in KEY-02-ROOTFS-EVIDENCE.md
```
**Claim.** The shipped rootfs tar contains no SSH host keys, no TLS private key, and no
machine-id — so no two nodes flashed from one image can share them.
**Gotcha, recorded because it will waste your afternoon:** read `RECIPE_HASH` from
`image-recipe/build/auto-installer/archipelago-rootfs.recipe.sha256`, **not** by hashing the
repo file. The wrapper rewrites 35 path expressions and absolutises `SCRIPT_DIR` before exec,
so the hash is host- and checkout-specific.
**Note the inverted expectation.** The original audit expected these artefacts to be *present*.
This check passes when they are *absent*.
---
## 6. Tier 3 — two physical nodes 🔒
**C-3 — host-key uniqueness.** Flash two machines from the *same* ISO, then compare:
```bash
# on each node
sha256sum /etc/ssh/ssh_host_*_key.pub
sha256sum /etc/ssl/private/<tls-key> # path per the nginx config
cat /etc/machine-id
```
Every value must differ between the two nodes. Any match is a finding.
SSH host keys and the TLS key are equally sharp signals once the single-producer work lands
(before it, TLS had an installer fallback and SSH did not — see `KEY-02-ROOTFS-EVIDENCE.md`).
---
## 7. Tier 4 — pre-release gate
```bash
# ON the node, not over RPC — it uses local podman/systemctl/bitcoin probes
ARCHY_ITERATIONS=5 bash tests/lifecycle/run-gate.sh
```
Install / UI / stop / start / restart / reinstall / reboot-survive /
archipelago-restart-survive / uninstall, 5× green. See `tests/lifecycle/TESTING.md`.
Frontend: `cd neode-ui && npm run test` (vitest) and `npm run build`.
Rust: `cd core && cargo test -p archipelago`.
---
## 8. Known-accepted risks
Recorded so an auditor does not have to discover them by reading commit messages.
| Risk | Decision | Where |
|---|---|---|
| A node whose first-boot secret generation can never succeed will not serve TLS | Accepted. Mitigated by a build-time assertion on generator binaries, retry-with-backoff, and self-heal on subsequent boots — leaving genuinely-broken hardware as the residual | `10-03` |
| Rotating host keys on already-deployed nodes invalidates `known_hosts` fleet-wide | Accepted, rated one-way, gated behind a decision checkpoint | D-06, `10-04` |
| KEY-01's fix ships on the next scheduled OTA, not an emergency release | Deliberate. The Critical finding stays live on the fleet until that OTA | D-10 |
| `#[cfg(test)]` code implements `rand::CryptoRng` falsely | Accepted until KEY-05 retires it; contained to test builds | `seed.rs:656` |
---
## 9. Reporting a finding
If any check above fails, or you find something not covered: the audit format that produced this
work is `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` — evidence as `file:line`, an explicit
severity, and a stated confidence. Findings that cannot be verified without hardware belong in an
UNVERIFIED section rather than being asserted.
Two corrections in that document are worth reading as calibration, because both were ours: F-10
**understated** its scope by a factor of 20, and the correction to it then **overstated** the
severity of two files within a day. Both are struck in place rather than rewritten.
docs/security/PSBT-SIGNING-ARCHITECTURE.md
+6 -8
View File
@@ -23,7 +23,7 @@
> - **§5 (LND) is unaffected and remains accurate**, including **§5.4's honesty table**, which is
> correct as written and unchanged.
>
> **For the current state, read `docs/security/KEY-03-SIGNING-POSTURE.md`** — it records the
> **Note:** the current signing-posture record is maintained internally. It records the
> deletion with its evidence, an honest per-step coverage map of the LND PSBT round trip, and the
> verdict on whether an external signer can sign a default node's PSBT today (it cannot: no fleet
> node is provisioned watch-only). Phases 2-7 below are unaffected as design targets.
@@ -32,15 +32,14 @@
> a phased rollout that a future `/gsd-plan-phase` can consume directly. It deliberately
> contains no code, adds no dependencies, and changes no wallet or signing behaviour.
>
> **Companion document:** `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` — the entropy and
> seed-generation audit that motivated this spec. **Cross-linked design:**
> **Companion document:** the internal entropy and seed-generation audit that
> motivated this spec. **Cross-linked design:**
> `docs/hardware-signer-design.md` — the exploratory TROPIC01 air-gapped signer, which this
> architecture treats as the future *first-party* signer, not as a competing design.
**Provenance rules used throughout.** Every architectural claim is grounded in either (a) a
`file:line` from this tree, or (b) RESEARCH.md Part C
(`.planning/quick/260731-upz-research-coinkite-conkite-low-entropy-ha/260731-upz-RESEARCH.md`,
which cites Bitcoin Core `doc/psbt.md`, `doc/descriptors.md`, `doc/multisig-tutorial.md`, the
(which cites Bitcoin Core `doc/psbt.md`, `doc/descriptors.md`, `doc/multisig-tutorial.md`, the
Core 30.0 release notes, LND `docs/remote-signing.md` and `docs/psbt.md`). Anything from
neither is marked `[UNVERIFIED]`.
@@ -426,7 +425,7 @@ nagged-at that users stop reading warnings. Concretely:
- **A software fix does not repair an already-generated seed.** If a seed was produced by a
defective RNG, updating the software leaves it exactly as guessable. This is why Coinkite told
users to migrate rather than merely update.
- **The audit found no such defect in Archipelago.** `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md`
- **The audit found no such defect in Archipelago.** The internal entropy audit's
§2 and §4 record that every first-party key-generation call site draws from a genuine CSPRNG,
that the mnemonic is a real 256-bit value, and that `[ARCHY-1]` is a *structural* risk with no
present exploitability.
@@ -610,11 +609,10 @@ current security posture and should not wait for the rest.
## 9. Related documents
- `docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` — the audit motivating this spec; see F-05
- The internal entropy and seed-generation audit — motivating this spec; see F-05
(Argon2 parameters) and the F-13 addendum on the xprv-in-Core issue.
- `docs/hardware-signer-design.md` — the first-party TROPIC01 air-gapped signer; §4.3 above
answers two of its open items.
- `docs/adr/005-chacha20-backup-encryption.md` — the at-rest envelope §6 reuses.
- `.planning/quick/260731-upz-research-coinkite-conkite-low-entropy-ha/260731-upz-RESEARCH.md`
— Part C is the source for the Core RPC table, the LND capability matrix, and the air-gap
format comparison.
docs/troubleshooting.md
+72 -28
View File
@@ -1,6 +1,6 @@
# Archipelago Troubleshooting Guide
This guide covers the 20 most common issues you may encounter with Archipelago, along with diagnostic commands and solutions.
This guide covers the most common issues you may encounter with Archipelago, along with diagnostic commands and solutions.
## Connection & Access
@@ -46,7 +46,11 @@ curl -s -X POST http://localhost:5678/rpc/v1 \
```
**Solutions**:
- Default password is `password123` — change it after first login
- There is no default password — the password is the one you created on this
node's first-boot "Set Up Your Node" screen. Password recovery requires SSH
access to the node; note that simply deleting `/var/lib/archipelago/user.json`
does **not** work, because the onboarding gate refuses `auth.setup` once the
node is provisioned
- Clear browser cookies and try again (stale session cookie)
- Restart the backend: `sudo systemctl restart archipelago`
- Check if the database is accessible: `ls -la /var/lib/archipelago/`
@@ -107,9 +111,12 @@ df -h /var/lib/archipelago
**Solutions**:
- If the image is missing: reinstall the app from the Marketplace
- If disk is full: run disk cleanup from Settings, or manually `podman system prune`
- If disk is full: run disk cleanup from the **Server** page (`/server`), or manually `podman system prune`
- If the container exits immediately: check logs for the root cause (usually missing config or permissions)
- Restart podman: `sudo systemctl restart podman`
- Restart the Podman socket. Archipelago runs **rootless** Podman as the
`archipelago` user, so this is a `--user` unit — `sudo systemctl restart podman`
would restart the unrelated root socket:
`systemctl --user restart podman.socket`
### 6. App shows "unhealthy" status
@@ -142,20 +149,34 @@ podman logs <app-id> --tail 100 | grep -i error
# Check Bitcoin logs
podman logs bitcoin-knots --tail 50
# Check if Bitcoin is connected to peers
podman exec bitcoin-knots bitcoin-cli -datadir=/data getpeerinfo | grep -c '"addr"'
# Check if Bitcoin is connected to peers.
# The datadir inside the container is /home/bitcoin/.bitcoin, and the RPC
# credentials live in the generated /tmp/rpc.conf (the manifest's entrypoint
# writes it from the BITCOIN_RPC_USER/BITCOIN_RPC_PASS secrets) — bitcoin-cli
# needs both flags or it can't authenticate.
podman exec bitcoin-knots bitcoin-cli \
-datadir=/home/bitcoin/.bitcoin -conf=/tmp/rpc.conf \
getpeerinfo | grep -c '"addr"'
# Check sync progress
podman exec bitcoin-knots bitcoin-cli -datadir=/data getblockchaininfo | grep -E "blocks|headers|verificationprogress"
podman exec bitcoin-knots bitcoin-cli \
-datadir=/home/bitcoin/.bitcoin -conf=/tmp/rpc.conf \
getblockchaininfo | grep -E "blocks|headers|verificationprogress"
```
**Solutions**:
- Initial sync takes 1-7 days depending on hardware — be patient
- Ensure the server has a stable internet connection
- Check disk space: Bitcoin requires 600GB+ for full chain
- Check disk space. The manifest picks the mode from the disk it's given: under
1000 GB it runs **pruned** (`-prune=550`, a few GB); at 1000 GB or more it runs
a full `-txindex=1` archival node, which needs 600 GB+ and growing
- If stuck: restart the container `podman restart bitcoin-knots`
- If peers = 0: check firewall allows port 8333 outbound
- Add manual peers: edit bitcoin.conf to add `addnode=` entries
- Editing `bitcoin.conf` in the data directory has **no effect** — the
entrypoint runs bitcoind with an explicit `-conf=/tmp/rpc.conf` and logs
"ignoring legacy datadir bitcoin.conf". Flags come from the app manifest, so
persistent changes belong there (and, for catalog-covered apps, in the signed
catalog entry that overrides the on-disk manifest)
### 8. LND won't connect to Bitcoin
@@ -250,7 +271,9 @@ df -h /
- Ensure at least 2GB free disk space
- If update hangs: wait 10 minutes, then restart the backend
- Do NOT power off during an update — this can corrupt the system
- If system is in a bad state after failed update: boot from the USB installer and select "Repair"
- If the system is in a bad state after a failed update, recover over SSH — the
USB installer has no repair mode (its boot menu offers only "Install
Archipelago", "Install Archipelago (verbose)" and "Boot from local disk")
### 12. Server won't boot after update
@@ -259,8 +282,11 @@ df -h /
**Solutions**:
- Wait 5 minutes — the first boot after update may take longer
- If still unresponsive: connect a monitor/keyboard to check boot messages
- Try the recovery mode: boot from USB installer and select "Repair"
- As a last resort: reflash the USB and restore from backup
- If it's a bootloader problem rather than a disk problem, boot the USB and pick
"Boot from local disk" to chainload the installed system
- As a last resort: reinstall from USB and restore from backup. The installer is
interactive — it asks for the target disk and requires typing `yes` — so
booting it does not by itself destroy the existing install
---
@@ -308,21 +334,35 @@ xrandr --query 2>/dev/null || echo "No display server"
**Symptoms**: Settings shows "Tor: Not configured" or the .onion address is missing
Tor is **not** a container — it's the host's Debian `tor` package, running as
`debian-tor`. Archipelago never touches it directly: it stages a torrc and asks
`archipelago-tor-helper` (a `.path` unit watching
`/var/lib/archipelago/tor-config/tor-action`) to install it and restart Tor.
**Diagnosis**:
```bash
# Check Tor container
podman ps --filter "name=tor"
podman logs tor --tail 20
# Check the host Tor service and the helper that drives it
sudo systemctl status tor
sudo journalctl -u archipelago-tor-helper --since "10 minutes ago"
# Check if Tor hostname file exists
cat /var/lib/archipelago/tor/hidden_service/hostname 2>/dev/null
# Is the SOCKS port up? (this is the liveness check the backend itself uses)
nc -z 127.0.0.1 9050 && echo "Tor SOCKS OK"
# The readable hostname copy the backend actually reads
cat /var/lib/archipelago/tor-hostnames/archipelago
# The hidden-service dir itself (root-owned 0700 — needs sudo)
sudo cat /var/lib/tor/hidden_service_archipelago/hostname 2>/dev/null \
|| sudo cat /var/lib/archipelago/tor/hidden_service_archipelago/hostname
```
**Solutions**:
- Tor takes 30-60 seconds to bootstrap — wait and refresh
- If Tor container is stopped: start it from the Apps page
- Check that the Tor data directory exists and has correct permissions
- Restart Tor: `podman restart tor`
- If `/var/lib/archipelago/tor-hostnames/archipelago` is missing but the
hidden-service dir has a `hostname`, the readable copy didn't sync — the
helper's `sync-hostnames` action rewrites it
- Check that the Tor data directory exists and is owned by `debian-tor`
- Restart Tor: `sudo systemctl restart tor`
### 16. Peers can't reach my node
@@ -330,11 +370,11 @@ cat /var/lib/archipelago/tor/hidden_service/hostname 2>/dev/null
**Diagnosis**:
```bash
# Check if Tor is running (needed for peer connectivity)
podman ps --filter "name=tor"
# Check if Tor is running (the fallback transport for peer connectivity)
sudo systemctl status tor
# Check your Tor address
cat /var/lib/archipelago/tor/hidden_service/hostname
cat /var/lib/archipelago/tor-hostnames/archipelago
# Test connectivity from the server side
curl -s http://localhost:5678/rpc/v1 \
@@ -343,10 +383,13 @@ curl -s http://localhost:5678/rpc/v1 \
```
**Solutions**:
- Ensure Tor is running (required for peer-to-peer communication)
- Tor is the last-resort transport, not the only one: peering prefers mesh
radio, then LAN, then FIPS, and only falls back to Tor. A peer stuck on
"unreachable" with Tor healthy usually means the higher transports are all
down too — check the FIPS anchor first
- Tor circuits can be slow — connections may take 30+ seconds
- Share your correct .onion address with peers
- Both nodes must have Tor running and be on the same federation
- Both nodes must be on the same federation
### 17. DNS resolution issues
@@ -366,7 +409,7 @@ podman exec bitcoin-knots nslookup seed.bitcoin.sipa.be
```
**Solutions**:
- Configure DNS from Settings > Network: try Cloudflare (1.1.1.1) or Google (8.8.8.8)
- Configure DNS from the **Server** page (`/server`): try Cloudflare (1.1.1.1) or Google (8.8.8.8)
- If using custom DNS, verify the server addresses are correct
- Restart networking: `sudo systemctl restart systemd-resolved`
@@ -414,7 +457,7 @@ podman system df
```
**Solutions**:
- Run disk cleanup from Settings
- Run disk cleanup from the **Server** page (`/server`)
- Remove unused app data: `podman system prune -a` (WARNING: removes all stopped containers and unused images)
- Move Bitcoin data to external drive if chain data is too large
- Check for large log files: `du -sh /var/log/*/ | sort -rh`
@@ -556,7 +599,8 @@ If the system is completely unresponsive:
1. **Power cycle**: Hold power button for 10 seconds, then turn back on
2. **Wait 5 minutes**: Services take time to start, especially if containers need to recover
3. **SSH in**: If web UI is down but SSH works, restart services manually
4. **USB recovery**: Boot from the Archipelago USB installer and select "Repair"
4. **Chainload the installed system**: Boot the Archipelago USB and pick "Boot
from local disk" — this rules out a broken bootloader
5. **Clean install + restore**: As last resort, do a fresh install and restore from backup
### Collecting Diagnostic Information
docs/user-walkthrough.md
+13 -5
View File
@@ -91,13 +91,21 @@ The auto-installer handles everything:
2. Tap or click anywhere to proceed
3. A typing animation welcomes you: "Welcome, Noderunner"
### Step 8: Login Screen
### Step 8: Create Your Password
> **Screenshot**: The login screen with a password field and glass-morphism design.
> **Screenshot**: The "Set Up Your Node" screen with password and confirm-password fields, glass-morphism design.
1. Enter the default password: `password123`
2. Click "Login"
3. You'll be prompted to change this password immediately
**There is no default web password.** A freshly installed node has no user
account at all, so this screen shows a password-creation form rather than a
login form:
1. Enter a password (minimum 8 characters)
2. Confirm it in the second field
3. Click "Set Up Node"
Every boot after this one shows the normal login form and asks for the password
you chose here. Store it somewhere you can get back to — recovering it requires
SSH access to the node.
### Step 9: Choose Your Path (Onboarding)
docs/workstream-b-signing-runbook.md
+18 -2
View File
@@ -1,12 +1,28 @@
# Workstream B — Signed app-catalog: completion runbook
**Status (2026-06-28):** The registry-distributed manifest pipeline is live — nodes fetch
**Status: ✅ COMPLETE** (runbook retained for re-running the ceremony — key
rotation, a new publisher, or a fresh release root).
The ceremony described below has been performed. Verified 2026-08-08:
- The anchor is **pinned** — `trust::anchor::RELEASE_ROOT_PUBKEY_HEX` is a
`Some(...)`, not `None`.
- `releases/app-catalog.json` carries a `signature` and a `signed_by` did:key.
Everything below therefore describes how to *do* the ceremony, not work that is
outstanding. The one-way-door warning in "Why this is gated on you" still
applies in full to any re-run: once a binary pins an anchor, a catalog signed by
a different key is hard-rejected fleet-wide.
---
**Original status (2026-06-28):** The registry-distributed manifest pipeline is live — nodes fetch
`releases/app-catalog.json` from the OTA mirror and embed manifests (origin-wins, disk
fallback). What remains for Workstream B is **authenticity**: pin the release-root anchor and
ship a *signed* catalog so nodes can cryptographically verify the publisher.
Today the catalog is **accepted unsigned** ("migration window") and the anchor is **unpinned**
(`core/archipelago/src/trust/anchor.rs:21` → `RELEASE_ROOT_PUBKEY_HEX = None`). Completing B is
(`core/archipelago/src/trust/anchor.rs` → `RELEASE_ROOT_PUBKEY_HEX = None`). Completing B is
a coordinated ceremony that **only the publisher can run** — it needs the offline
`RELEASE_MASTER_MNEMONIC`, which is not (and must not be) stored on any node or build host.