archy/docs/1.8.0-RELEASE-HARDENING-PLAN.md

# Archipelago 1.8.0 — Release Hardening Plan & Tracker

> **The one living checklist for shipping 1.8.0.** Derived from a full-system deep
> audit (2026-07-02): backend security, backend code-quality, frontend, mesh,
> tests/release pipeline, and the ISO build. Supersedes nothing — it *sits above*
> `docs/UNIFIED-TASK-TRACKER.md` (day-to-day) as the release exit-criteria list.
> **Keep it updated: tick a box the moment an item lands, with the commit sha.**

**Definition of done for 1.8.0:** the supply chain is authenticated end-to-end
(§A), OTA self-update is safe and rollback-proven on real hardware (§B), no
secrets ship in the image (§F), and the single-node gate stays 5/5 green through
all of it. Everything else is polish that should not block the tag.

**Legend:** `[ ]` open · `[~]` in progress · `[x]` done · 🔴 critical · 🟠 high ·
🟡 medium · 🟢 low/polish · ⛔ blocked on you.

---

## 🎯 The single most important insight

The **release signing ceremony (Workstream B) is the linchpin.** ✅ The ceremony
KEY was generated (user confirmed 2026-07-02) — the hard offline part is done. But
the outputs are **not yet wired into the repo**: `anchor.rs:21` is still `None` and
`releases/app-catalog.json` carries no `signature`/`signed_by` (its `image_signature`
fields are literal `"cosign://..."` placeholders). Three mechanical steps remain,
split by who can run them: **(1)** pin the pubkey — needs only the *public* hex, can
be done in-repo now; **(2)** sign the catalog with the `RELEASE_MASTER_MNEMONIC` —
only the publisher, secret never touches a host; **(3)** implement + flip cosign
enforcement on the pull path. Until (1)+(2) land, every "verify the signature" task
below is written but not enforced. **This is still the critical path; §A converges on it.**

---

## §A — Supply-chain authentication (🔴 THE release blocker)

Today an attacker who controls the mirror IP (or any MITM on the plaintext HTTP
path) can ship an arbitrary root binary, arbitrary container images, and an
arbitrary app catalog to the entire fleet — fully unattended under
`auto_apply`. These four items are one story and must land together.

- [x] 🔴 **Pin `RELEASE_ROOT_PUBKEY_HEX` + sign the catalog** — DONE 2026-07-02.
      `anchor.rs` pinned to `5d15cbee…d469951` (signer
      `did:key:z6MkkidEnEpo6qHMCNSZoNKWtvQvxq3whnaME9wGgEFhq7ur`); trust tests updated (16/16
      green). `releases/app-catalog.json` signed in place (`signed_by` matches, 64-byte sig);
      two blocking floats fixed en route (`archy-btcpay-db` version→string, `cpu_limit` 0.25→1).
      Ship order (backward-compatible): signed catalog goes out first (old binaries still accept
      it), pinned-anchor binary follows in the next build/OTA. **Still ahead:** (a) the
      pinned-anchor binary must actually be built + shipped for enforcement to be live on nodes;
      (b) flip "accept unsigned" → "reject unsigned" only after the whole fleet is on the pinned
      binary (`container/app_catalog.rs:397`, the `Unsigned` arm) — see the next item.
- [ ] 🔴 **Enforce a signature on the OTA manifest before trusting it.**
      `update.rs:68` fetches `http://146.59.87.168:3000/.../manifest.json` over cleartext
      and parses/trusts it with no `trust::verify_detached` call; component sha256/blake3
      are only checked against that same unauthenticated manifest → remote root RCE.
      Move to HTTPS + pinned cert, require an Ed25519 release-root signature, and
      **refuse `auto_apply` until the anchor is pinned.**
- [ ] 🔴 **Implement container image signature verification (cosign).**
      `container/src/podman_client.rs:255` — `pull_image(.., _signature)` silently discards
      the signature that the manifest threads all the way down
      (`prod_orchestrator.rs:1978/2435`). Wire `sigstore-rs`/`cosign verify` (or
      `podman pull --signature-policy`); hard-fail when a declared signature doesn't verify.
- [ ] 🟠 **Move the image mirror to HTTPS; drop `--tls-verify=false`.**
      `podman_client.rs:641` `INSECURE_REGISTRY_HOSTS = ["146.59.87.168:3000"]` +
      `config.rs:104,124` allowlist pull images over unauthenticated HTTP. Remove the raw-IP
      entries; give the mirror a valid/pinned cert. (Same host also baked insecurely into
      the ISO — see §F.)
- [ ] 🟠 **Validate every image string at the pull site, not just the RPC boundary.**
      `is_valid_docker_image` runs in `install.rs:224`/`runtime.rs:549` but
      `prod_orchestrator::install_fresh` (1978) and `resolve_catalog_image` (944-971) pass
      catalog/manifest images straight to `pull_image`. Call the validator right before
      every pull.

---

## §B — OTA self-update safety (🔴 1.8.0's headline feature is untested live)

The apply path itself is well-built (resumable download, staged-complete marker,
atomic swap, single-depth backup). The gaps are **authenticity** (§A) and
**verification depth** — plus the fact that the upgrade path has never run
end-to-end on real hardware.

- [ ] 🔴 **Deepen the post-OTA health check.** `update.rs:456` (`probe_frontend_once`)
      passes on any 2xx/3xx from `GET /`, and `verify_pending_update` (494-593) only rolls
      back on that. A release with a broken RPC API, dead containers, or failed LND unlock
      passes and never rolls back. Add `/rpc/v1 update.status` + container-list/required-stack
      health assertions before clearing the pending-verify marker.
- [ ] 🟠 **Run one real upgrade-from-vN-1 soak on hardware before tagging.**
      No test installs the previous version, points it at a staged 1.8.0 manifest, applies,
      and asserts health + rollback. This is the top release risk for an OTA release. A
      two-VM (or two-node) harness is enough.
- [ ] 🟡 **Guard the frontend-build-no-op in the *actual* release path.** The
      `ui-dist-version` grep guard (`tests/release/run.sh:82`) is behind `--with-build`, which
      `scripts/create-release.sh:90` never passes → a stale frontend can ship with a valid
      sha256. Call `run.sh --with-build --manifest` from create-release (or fold the grep in).
- [ ] 🟢 **publish-release-assets verifies size, not sha256** (`publish-release-assets.sh:97`).
      Add a HEAD/GET sha256 compare so a size-correct/content-wrong mirror asset fails the
      publish gate.

---

## §C — Backend robustness (🟠 stability, mostly low-effort/high-ROI)

Note: the `.unwrap()`/`panic!` worry is a **non-issue** — nearly all are in test
modules; production request/boot paths are essentially panic-free. The real risks:

- [ ] 🟠 **Log swallowed persistence writes.** ~30-40 dangerous `let _ = save_*().await`
      sites discard durability failures with zero diagnostics: `server.rs:270` (mesh config),
      `bitcoin_relay.rs:865` (relay state), `update.rs:163/1223` (mirrors/update state),
      `registry.rs:158`, `mesh/status.rs:286`, `scheduler.rs:179`, `install.rs:34`. Convert to
      `if let Err(e) = … { warn!(…) }`; leave genuinely fire-and-forget ones commented.
- [ ] 🟠 **Remove blocking `std::process::Command` from async handlers.**
      `install.rs:2222` `published_host_port` (sync podman on the install path),
      `dependencies.rs:316` (`df`), `system/handlers.rs:578` (`sudo`), `transport/fips.rs:50`
      (`systemctl`) stall tokio workers under load. Convert to `tokio::process` or
      `spawn_blocking`. Only 8 files use `std::process::Command` — bounded.
- [ ] 🟡 **Restrict Bitcoin RPC exposure.** `bootstrap.rs:409` writes
      `rpcallowip=0.0.0.0/0`. Scope to the container subnet / `127.0.0.1`.
- [ ] 🟡 **Move generated secrets from env to file mounts.** `manifest.rs:1208-1226`
      injects secrets as `-e KEY=value`, readable via `podman inspect` / `/proc/<pid>/environ`.
      Prefer bind-mounting the existing `0600` secret file or `podman --secret`.
- [ ] 🟡 **Harden rate-limit IP extraction.** `middleware.rs:120-128` trusts
      client-spoofable `X-Real-IP`/`X-Forwarded-For` → per-request bucket rotation defeats the
      login limiter. Trust forwarded headers only from a configured proxy; have nginx set them.
- [ ] 🟢 **Include `seq` in the mesh signed preimage.** `message_types.rs:245-288` signs
      `(t,v,ts)` but sets the anti-replay `seq` after signing → a radio MITM can alter ordering
      without breaking the signature.
- [ ] 🟢 **Guard the short-DID slice panic** (`mesh/listener/decode.rs:566`) and gate the
      dev-mode `password123` bypass (`auth.rs:18`) behind `#[cfg]` before it can reach a
      release build.
- [ ] 🟢 **Apply the seccomp/apparmor profile** — `security/src/container_policies.rs:71` is a
      TODO; the profile is defined but never applied to podman.

---

## §D — Frontend security & performance (🟠)

The untrusted mesh/LoRa chat path is **safe** (interpolation, no `v-html` — good).
The real issues are the app-bridge origin model and a bloated bundle.

- [ ] 🟠 **Validate `event.origin` + add consent gates in the NIP-07 nostr bridge.**
      `stores/appLauncher.ts:385-490` derives the caller from the launcher's own URL, never
      `event.origin`, and `getPublicKey`/`nip04.decrypt`/`nip44.decrypt` have no consent gate →
      any co-resident iframe can deanonymize the nostr identity or use the node as a decryption
      oracle while an app is open. Check `event.origin` against the open app's real origin; key
      approvals on it; gate decrypt/getPublicKey like `signEvent`.
- [ ] 🟠 **Origin-check the `share-to-mesh` handler.** `App.vue:450-464` acts on
      `{type:'share-to-mesh', cid}` from any sender and force-navigates to `/mesh` with the CID
      pre-staged. Add `ev.origin === window.location.origin` (as `Chat.vue:95` already does).
- [ ] 🟡 **Decide the app-iframe isolation model.** `AppSessionFrame.vue:54` /
      `AppLauncherOverlay.vue:79` embed apps same-origin with no meaningful `sandbox`; a
      same-origin app can read the CSRF cookie + `localStorage`. Ideal fix (serve apps from a
      per-app subdomain origin) is architectural — at minimum decide + document for 1.8.0.
- [ ] 🟡 **Shrink the 93 MB dist.** `assets/video/video-intro.mp4` is **14.7 MB**
      (precached by the service worker → blocks PWA install), plus ~18 MB of ~1 MB full-screen
      JPEGs. Convert backgrounds to WebP/AVIF at responsive sizes, lazy/stream the intro video,
      and exclude video/audio from the Workbox precache. Biggest, easiest perf win.
- [ ] 🟢 **DOMPurify the `Server.vue` QR SVG** (`:283/:295` render `v-html` unsanitized while
      `TwoFactorSection.vue` sanitizes the analogous SVG); guard the unguarded `pollInterval`
      (`Mesh.vue:391`); surface silent data-fetch failures (`curatedApps.ts:58/71`).

---

## §E — Mesh transports (🟢 mostly done — verify & polish)

Confirmed **fixed in HEAD:** B8 (1970 timestamps), B6 (inbound RX surfacing), the
per-message transport pill, and the archy↔archy plain-TEXT-DM E2E fix. Remaining:

- [ ] 🟠 **Active Reticulum daemon-death detection.** `reticulum.rs:589` only `warn!`s on
      socket EOF and `try_recv_frame` then returns `Ok(None)` forever; nothing calls
      `child.try_wait()`. On an idle link a crashed daemon is invisible for up to 30 min (the
      RX-stall timeout). Treat socket EOF as `Err` → immediate respawn. (Pairs with the current
      `fix/reticulum-daemon-pdeathsig` branch work.)
- [ ] 🟡 **Persist chat history across restarts.** `state.messages` boots empty
      (`listener/mod.rs:283`) while outbox/scheduler/peers survive — inconsistent; bubbles
      vanish on restart. Add `mesh-messages.json` mirroring the `scheduler.rs`/`outbox.rs`
      pattern (or explicitly accept the loss).
- [ ] 🟡 **Tighten the 30 s legacy dedup** (`listener/mod.rs:383-389`) — it silently drops a
      peer legitimately sending identical text twice within 30 s.
- [ ] 🟢 **Wire the PyInstaller daemon binary into the release tarball / deploy script**
      (Rust expects `/usr/local/bin/archy-reticulum-daemon`, `reticulum.rs:80`); add the RNode
      udev rule; finish `ARCHY:2:` announce→`arch_pubkey_hex` binding (`reticulum.rs:119`).
- [ ] 🟢 **Duty-cycle guard for LoRa TX** — none exists; EU 868 is legally 1%. At minimum an
      airtime budget/warning.

---

## §F — ISO / image build (🔴 one secret leak; otherwise 🟠 hardening)

`image-recipe/_archived/build-auto-installer-iso.sh` (3604 lines) is the real
builder; OTA is the normal update path but the ISO is what produces installable
media (latest artifact only one minor behind).

- [ ] ⛔🔴 **Anthropic API key — INTENTIONAL for alpha/beta, hard GO-LIVE gate.**
      `build-auto-installer-iso.sh:2645` bakes a live `sk-ant-…` key into `claude-api-proxy.service`
      so alpha/beta testers get frictionless AI (deliberate — per user 2026-07-02). **Do NOT
      remove for alpha/beta.** Before public GA it MUST be removed + rotated + injected at runtime
      (a second copy also exists in a worktree). Track it here so it can't be forgotten at launch.
- [ ] 🔴 **Per-device secrets on first boot.** The self-signed TLS **private key is generated
      at build time** (`:426`) → every device ships the same key; SSH host keys likewise not
      regenerated. Generate TLS + SSH host keys on first boot.
- [ ] 🟠 **Kill default credentials.** `archipelago`/`archipelago` (SSH+root), web `password123`,
      and SSH `PasswordAuthentication yes` (`:411`) all ship. Lock root, force credential
      creation in onboarding, disable SSH password auth (or force-change on first login).
- [ ] 🟠 **Sign + checksum the ISO.** Pipeline ends at `xorriso` with no `SHA256SUMS`, no
      GPG/minisign, no Secure Boot (`BOOTX64.EFI` is unsigned though `grub-efi-amd64-signed` is
      installed). Emit + sign checksums; wire signed Secure Boot.
- [ ] 🟠 **Registries over HTTPS in the image too** — `146.59.87.168:3000` / `git.tx1138.com`
      are baked `insecure=true`/`tls_verify:false` (`:216`, `:2308`). (Ties to §A.)
- [ ] 🟡 **Add `unattended-upgrades` + a default-deny nftables firewall** (allow 22/80/443 +
      mesh/WG). Neither exists today; OS packages drift until reflash and there is no host
      firewall.
- [ ] 🟡 **Pin the build for reproducibility.** FIPS daemon is built from unpinned upstream
      `main`, Tailscale from its live apt repo, and `scripts/image-versions.sh` uses many
      `:latest`/`stable` tags (+ `bitcoin-ui:1.7.84-alpha`, 15 behind). Pin to commits/versions;
      snapshot apt. Wire ISO version to `Cargo.toml` so it can't drift.
- [ ] 🟢 **Harden LUKS + roadmap A/B partitioning.** The LUKS data key sits in plaintext on the
      unencrypted root (`:2137`); add TPM2/passphrase binding. Longer-term: A/B (or
      factory-reset) partitions for safe OTA rollback, and a real install-time TUI
      (`docs/INSTALL-SCREENS-DESIGN.md` exists but the installer is headless "press Enter").

---

## §G — Refactor & code health (🟢 not release-blocking; do after the tag or opportunistically)

- [ ] 🟢 **Manifest-drive per-app special-casing.** App names are branched on across 5-7 Rust
      files (`config.rs` 36 match arms, `runtime.rs` 17, `install.rs:275-287` dispatch,
      `prod_orchestrator.rs:54-83` baseline/restart-sensitive lists). Move `baseline`,
      `restart_sensitive`, `stack_members`, `multi_container` into the manifest schema; collapse
      the five near-identical `install_*_stack()` wrappers into one generic call. **Biggest
      maintainability win.**
- [ ] 🟢 **Route all podman/systemctl through `podman_client`.** 113 raw `Command::new("podman")`
      + 32 `systemctl` calls bypass the existing 952-LOC wrapper → untestable + the blocking-call
      risk (§C). Consolidating also unlocks unit tests for the thinly-tested `package/` handlers
      (`stacks.rs` 1 test, `config.rs` 2, `runtime.rs` 3, `install.rs` 7).
- [ ] 🟢 **Split the god-modules.** `prod_orchestrator.rs` (5,263 LOC) → `orchestrator/{reconcile,
      host_ports,ownership,hooks}.rs`; `Mesh.vue` (2,485 LOC / 241 KB chunk) → sub-components.
      Both are well-tested, so safe.
- [ ] 🟢 **Delete dead code.** ~4,100 LOC of orphan StartOS crates (`js-engine`, `models`,
      `helpers`, `container-init`) not in the workspace or linked; the committed AppleDouble
      `._*.rs` files; the committed `.venv/`/`build/`/`__pycache__` under the duplicate
      `reticulum-daemon/` tree; promote `MeshRadioDevice` enum → trait.
- [ ] 🟢 **Resolve the Quadlet flag & dep hygiene.** Decide `use_quadlet_backends`' fate
      (flip default + delete the legacy `create_container` branch, or freeze as experimental —
      don't ship both half-maintained). Consolidate the mixed hyper 0.14/1.x ecosystem; bump
      stale majors (reqwest, base64, thiserror, tokio-tungstenite).

---

## §H — Testing gaps that gate confidence (🟠)

- [ ] 🟠 **Add the OTA upgrade soak** (same as §B item 2) — the highest-value missing test.
- [ ] 🟡 **Add a host-reboot survival tier** — every app is `○` (untested) for reboot in
      `TESTING.md:138`; the gate can't reboot the node it runs on. Run SSH-`reboot`-then-reprobe
      out-of-band per node.
- [ ] 🟡 **Make the release gate run the full Rust suite** (or hard-require a green CI sha).
      `tests/release/run.sh:101` runs only a 6-module slice because the full 1000-test suite
      hangs PTYs on the dev box → 994 tests unverified at release time if CI is stale.
- [ ] 🟡 **Add `--max-time` to `node_rpc()`** (`tests/multinode/lib/multinode.bash`) — a slow
      server-side RPC hangs the whole multinode suite with no feedback.
- [ ] 🟢 **De-hardcode creds/IPs in tests** (`tests/multinode/smoke.sh:32`,
      `remote-lifecycle.sh:136`); snapshot/restore node baseline between destructive iterations
      (teardown currently only clears `/tmp` session files).

---

## §I — Carried-over open items (from `UNIFIED-TASK-TRACKER.md`, still valid)

- [~] 🟠 **Multinode gate pass** — 5× destructive gate was launched on node `.5`; bring the
      rest of the fleet to precondition, then run the existing (undocumented-but-present)
      `tests/multinode/{smoke,meshtastic}.sh` cross-node suites.
- [ ] 🟠 **Federation `remove-node` tombstone regression.**
      `federation/storage.rs:187` does `let _ = tombstone_did(...)` — swallows the write error,
      so a removed peer reappears after the next sync. (This is a specific, confirmed instance
      of the §C swallowed-writes class.) Needs a careful fix + `smoke.sh` re-verify.
- [ ] 🟠 **Phase-3 Quadlet default-flip** — validated + opt-in on .228/.198; flip
      `config.rs:256` once the .5 gate reports clean.
- [ ] 🟠 **Developer CLI suite** (`archy app validate/render/install/test`) — gates external
      app publishing (`APP-PACKAGING-MIGRATION-PLAN.md` step 5).
- [ ] ⛔🟡 **Version-naming decision** (`1.7.99-alpha` → `1.8.0` vs `1.8.00-alpha`) — a one-line
      call, then a mechanical bump + tag. **Needs your decision.**
- [ ] ⛔🟢 **Bitcoin multi-version fleet OTA** — `.228` working on branch; rollout timing is
      held for your call (`docs/bitcoin-version-bulletproof-rollout.md`).
- [ ] ⛔🟢 **3ccc stock-Meshtastic RF validation** — code fix in place; needs a live radio send.

---

## Suggested order of attack

1. **The critical path:** §A signing ceremony → then turn on manifest/catalog/image
   signature enforcement (§A) + OTA HTTPS/signature + deeper health check (§B).
2. **Cheap high-ROI stability:** §C swallowed-writes + blocking-calls; §D nostr-bridge
   + share-to-mesh origin checks; §H OTA soak + reboot tier.
3. **Image hardening:** rest of §F (per-device secrets, default creds, ISO signing,
   firewall/unattended-upgrades, pinning).
4. **Polish, post-tag:** §G refactors, §E mesh persistence/dedup, §D bundle shrink.
5. **Decisions you own (⛔):** version name, signing mnemonic, bitcoin OTA timing, 3ccc test.
6. **Before public GA only (NOT alpha/beta):** remove + rotate the Anthropic key (§F) —
   intentionally left in for frictionless AI during alpha/beta.

*Last updated: 2026-07-02 (initial deep-audit synthesis). Update this line + tick
boxes with commit shas as items land.*