archy/docs/PRODUCTION-MASTER-PLAN.md

# 🚩 PRODUCTION MASTER PLAN — Archipelago App Platform & Registry

> **THIS IS THE AUTHORITATIVE PLAN. Agents: read this first and keep it open until
> the production test gate (§5) is green.** It overrides ad-hoc direction and
> supersedes all prior roadmap/handoff/status docs. When the gate passes, remove
> the priority banner and demote this doc.
>
> Last updated: 2026-06-22 · Binary: v1.7.99-alpha · See §8b for the live resume.

---

## 1. The North Star

Make Archipelago a **world-class, developer-ready app platform** where:

1. **Every app is manifest-driven** — install/run/update/uninstall needs only the
   app's manifest (+ catalog entry). **Zero OS-level code reliance**: no per-app
   Rust installers, no `sudo mkdir/chown`, no host provisioning.
2. **Manifests are distributed via the (signed) registry**, not baked into the
   binary OTA as disk files. Bumping/adding an app = a signed catalog change.
3. **Third-party developers can build and ship apps via an external registry** —
   a decentralized marketplace (DID-signed manifests, Nostr discovery, reputation),
   not a gatekept central store. `archy app validate/render/install/test` tooling.
4. The platform stays **rootless, secure-by-default, elegant, robust, and
   100%-uptime-capable** (reboot-survivable, self-healing, no data loss on migrate).

**Definition of done:** the production test gate (§5) is green for the app set on
real nodes. Until then, this plan is the priority.

## 2. Invariants (never violate)

- **Rootless Podman only.** No rootful, no Docker-socket mounts, no privileged
  containers unless explicitly approved. (ADR-001, ADR-009.)
- **No app-specific business logic in the Rust backend.** The orchestrator owns
  the lifecycle state machine; apps are declarative. Legacy `install_immich_stack`
  (hardcoded `podman run` + `sudo chown`) is the anti-pattern being deleted.
- **Secrets are manifest-declared** (`generated_secrets`, materialised by
  `container::secrets` 0600/rootless, idempotent + self-healing) — never hardcoded,
  per-app, or logged. Replaces the deleted `ensure_fmcd_password`.
- **Migrations never destroy data.** Preserve `/var/lib/archipelago/<app>`,
  generated secrets, displayed credentials, public ports, and adoption container
  names. Always provide a rollback path. Stop/recreate only when necessary.
- **Verify on a real node (.228, then .198) before any tag.**

## 3. Current state (2026-06-21)

- **~40 apps are manifest-based and Quadlet-migrated** (survive
  `archipelago.service` restart + reboot). Exhaustive per-app table:
  `docs/app-registry-status-2026-06-21.md`.
- **Legacy holdout: immich** — the one app with **no manifest** and a hardcoded
  Rust stack installer (in-cgroup, not Quadlet). 3 containers, healthy, live data.
  The migration proof case.
- **Manifests still travel by OTA disk rsync** (`apps/ → /opt/archipelago/apps`).
  The signed catalog (`app-catalog.json`) currently distributes **only image
  overrides** — not full manifests. Gap closed by workstream B.
- **The 4 companions** (`archy-bitcoin-ui`, `-lnd-ui`, `-electrs-ui`,
  `-fedimint-ui`) build from `docker/<name>` contexts via `companion.rs`, not the
  manifest registry — a later phase folds them in.
- **No app has passed the formal production gate (5× for now, was 20×).** That is the blocker.

## 4. Workstreams (each links its authoritative detail doc)

| # | Workstream | Detail doc | Status |
|---|-----------|-----------|--------|
| A | **Manifest-driven app platform** — packaging contract, single/multi-container runtime, routing, controlled hooks, dev tooling (6 phases, security model, migration rules) | `APP-PACKAGING-MIGRATION-PLAN.md` | mostly done; immich + multi-container polish remain |
| B | **Registry-distributed manifests** — catalog carries full signed manifest; orchestrator installs from registry; disk = migration fallback | `registry-manifest-design.md` | **phases 1+2 done** (node consume + opt-in publisher embed); not yet flipped on for the fleet |
| C | **Developer-ready external registry** — 3rd-party DID-signed manifests, decentralized Nostr discovery (NIP-78 kind 30078) + trust score, `archy app …` tooling | `marketplace-protocol.md`, `app-developer-guide.md` | design exists; tooling + trust UX pending |
| D | **Distribution backbone** — signed catalog, BLAKE3 content-addressing, iroh swarm (origin-always-wins) | `dht-distribution-design.md` | phases 0–2 code-complete (worktree) |
| E | **Production test gate** — 5× lifecycle on .228 + .198 (for now; was 20×), per-app L1/L2 matrix | `tests/lifecycle/TESTING.md`, `bulletproof-containers.md` | **never green — exit criterion** |

**Orchestrator architecture** (foundation for A/B): `rust-orchestrator-migration.md`
(ProdContainerOrchestrator, BootReconciler 30s level-triggered reconcile, adoption
scan, Quadlet rendering) and `bulletproof-containers.md` (the six container failure
modes FM1–FM6 + the desired-state-first reconciler that fixes them).

## 5. Production test gate (exit criterion)

An app is **production-ready** only when `tests/lifecycle/run-20x.sh` is green
across the full matrix — install / UI-reachable / stop / start / restart /
reinstall / **reboot-survive** / **archipelago-restart-survive** / uninstall —
**5× on .228 AND .198 for now** (`ARCHY_ITERATIONS=5`; temporarily reduced from
20× — restore to 20× before the final ship). All 8 gate checkboxes in `tests/lifecycle/TESTING.md`
are currently unchecked. Coverage today: L0 unit (631 ●), L1 RPC ● for 6 core apps,
L2 UI ● dashboard + proxies; L3 survival ◐; ~30 apps have zero automated coverage.

## 6. Immediate sequence (live workstream)

1. ✅ **B-phase 1** — `manifest` field on `AppCatalogEntry`; `load_manifests`
   catalog-wins merge; `manifest_dir` kept (build-source catalog manifests skipped
   in phase 1); unit tests. *(commit 220666d3)*
2. ✅ **B-phase 2** — `EMBED_MANIFESTS` publisher generator + round-trip guard.
   *(7bfbe8fe; signing via existing ceremony — not yet flipped on for the fleet.)*
3. ✅ **C immich proof** — immich is a manifest-driven stack (immich + immich-postgres
   + immich-redis) installed via `install_stack_via_orchestrator`; legacy installer
   is now fallback-only. Live-migrated + verified on .228. Found+fixed: container_name
   duplicate-on-shared-PGDATA, version-digit validation, partial-fallback hardening,
   data_uid 100998. Canonical app_id `immich` (title+icon). *(9e6c5370, d5ef4573)*
4. ✅ **Reboot-survival** — podman-restart.service enabled (startup, fleet-wide)
   for the podman-`--restart` path. *(f160e0c4)*
5. ◻ **Verify on .198** (immich migration validated on .228 only so far).
6. ◻ **E** — run the 5× gate (`ARCHY_ITERATIONS=5`, was 20×); fix until green.
7. ◻ Demote this banner.

**Not yet done / deliberate follow-ups:** flip `EMBED_MANIFESTS` on for the
published catalog (then sign) to actually distribute manifests via the registry;
Phase-3 `use_quadlet_backends` rollout so orchestrator backends are Quadlet (not
just podman-`--restart`); immich on .198.

## 7. Release blockers & operational gotchas (durable)

Carried forward from prior handoffs (deduped against persistent memory):

- **Rootless control-plane responsiveness** — slow `podman ps`/store cleanup at
  startup must not surface a false "no apps installed" UI. **My Apps must preserve
  last-known apps during scanner backoff**, never show empty during a transient.
- **Reboot survival** — gate on ≥3 (prefer 5) consecutive clean post-reboot
  lifecycle passes. Quadlet units under `user.slice` survive `archipelago.service`
  restart; legacy in-cgroup containers get SIGKILLed and reconciled back.
- **Startup patterns** — wait on a socket/health, never `sleep`. Tailscale waits
  for its socket; Fedimint Guardian waits for Bitcoin RPC `initialblockdownload:false`
  before launching fedimintd (proxy/wait companion on :8175 during IBD).
- **Bitcoin must run full** (`txindex=1`, non-pruned) for ElectrumX/mempool.
- **Adoption** — match existing containers by name and adopt without recreate;
  record a migration version in app state; preserve Nostr signer bridges
  (IndeeHub needs `/nostr-provider.js` served, not just port reachability).
- **Image presence** — use bounded targeted `podman image inspect`, not
  `podman image exists` (avoids store-walk stalls).
- **Companion rebuilds** — `companion.rs` must rebuild `:latest` when the build
  context changes (staleness check), else baked-in fixes (e.g. guardian CSS) never
  reach nodes. `:local` is a manual override, never auto-rebuilt.

## 8. Roadmap

**Pipeline:** Feature Testing (internal) → User Testing (controlled hardware) →
Beta Live (public). Hardening priorities feeding the gate:

- **P0** Container app reliability — bulletproof install/health/restart/uninstall
  across all apps, dependency chains, multi-container stacks.
- **P0** Networking stack first-install → reboot-proof (WireGuard/NetBird, Tor
  hidden services, LND Connect).
- **P1** LUKS2 full-partition encryption for `/var/lib/archipelago/`
  (AES-256-XTS, Argon2id, key from setup password + hardware salt).
- **P1** Meshtastic plug-and-play parity with MeshCore.

**Post-beta (deferred — do not start until gate is green):** P2P encrypted
voice/video (WebRTC over federation via Tor); watch-only wallet + mesh BTC
hardening; paid swarm streaming + IndeeHub source (`phase4-streaming-ecash-plan.md`);
Meshroller Rust-native mesh AI (`meshroller-integration-design.md`); dual-ecash
phases 2–6 (`dual-ecash-design.md`).

## 8b. SESSION STATE + RESUME (updated 2026-06-22) — READ THIS FIRST ON RESUME

### Where we are — Task #20 (manifest lifecycle hooks) + indeedhub migration: DONE & 2-node verified

Manifest-driven lifecycle hooks + the IndeedHub stack migration are **complete and
live-verified on BOTH .228 and .198** (adoption + fresh-create + post_install hook
exec, stable under load). 15 commits this session: `4c1a4e59`..`e2a012d0`. Working
tree clean. The release lifecycle gate is temporarily **5×** (was 20×; `ARCHY_ITERATIONS=5`).

**Shipped (all on `main`, newest first):**
- `e2a012d0` indeedhub frontend health → `tcp:7777` (was http GET `/`; the http check
  false-failed under load and the reconciler churned the frontend — fixed).
- `ff78b312` hook `exec` runs in a transient user scope
  (`systemd-run --user --scope --quiet --collect podman exec …`) — fixes
  "crun: write cgroup.procs: Permission denied" when exec'ing from archipelago.service.
- `ff8f11b8` indeedhub frontend caps `[CHOWN,DAC_OVERRIDE,SETGID,SETUID]` — nginx
  workers died "setgid(101) failed" under the orchestrator's `--cap-drop=ALL`.
- `b73084db` DELETED the legacy indeedhub orchestrator special-cases (−382 lines:
  reconcile_indeedhub_stack, start_indeedhub_backends, the 120s dependency-DNS gate,
  patch_indeedhub_nostr_provider, repair_indeedhub_network_aliases, INDEEDHUB_* consts)
  → "indeedhub" now uses the GENERIC install_fresh/reconcile path.
- `b1eea8c0` 7 indeedhub manifests (apps/indeedhub{,-postgres,-redis,-minio,-relay,-api,
  -ffmpeg}) + `install_indeedhub_stack` orchestrator-first (immich pattern).
- `b94b61f6` `network_aliases` ContainerConfig field (podman_client + quadlet rendering,
  DNS-label validated) — lets the frontend nginx reach `api:4000`/`minio:9000`/`relay:8080`
  on the dedicated `indeedhub-net`.
- `955c54b7`/`4c1a4e59` #20 hooks phases 1-2: schema (LifecycleHooks/HookStep/HostCopy in
  archipelago-container::manifest) + executor `container::hooks::run_post_install`
  (allowlist-canonicalised copy_from_host + scoped exec), wired into `install_fresh`.
- `84031e62` gate 20×→5× (docs only: CLAUDE.md, this file, tests/lifecycle/TESTING.md).

**Design = adoption-safe + manifest-driven.** Manifests reproduce the live install exactly
so existing nodes ADOPT (NoOp) instead of recreate: hyphen container_names the runtime
already references, named volumes `indeedhub-{postgres,redis,minio,relay}-data`,
`indeedhub-net` + network_aliases [postgres|redis|minio|relay|api], generated_secrets reuse
the live /var/lib/archipelago/secrets values (ensure_one no-ops on existing; postgres pw is
fixed at PGDATA init). minio user "indeeadmin" + AES_MASTER_SECRET literal kept. The
frontend image indeedhub:1.0.0 already bakes the iframe nginx (X-Frame omit + nostr-provider.js
+ sub_filter), so the post_install hook (sed X-Frame / copy nostr-provider.js / inject /
nginx reload) is defensive/idempotent. crash_recovery.rs's frontend-after-deps ordering
guard is KEPT on purpose (beneficial; not a blocker).

### ⛔ GATE BLOCKER discovered 2026-06-22 — `package.stop` does not stop several apps (FIX FIRST)

Step 1 (sync .228 tcp-health manifest) is **DONE + verified** (frontend adopted, UI 200, no
churn). Step 2 (the 5× gate) surfaced a **real, reproducible blocker** — this is now the top task.

**Symptom.** On a CLEAN, healthy .228, `package.stop electrumx` returns `{"status":"stopping"}`
but the container **never stops** — `container-list` shows `running` for 66s+, the scanner keeps
logging `Detected container: ElectrumX (running)`. The gate's `wait_for_container_status electrumx
stopped 60` therefore times out. Same failure hit bitcoin-knots / btcpay-server / fedimint / immich
in the gate run. **Contrast:** `filebrowser` stops correctly (`running → stopped` in ~6s).

**Root-cause chain (evidence-backed, not fully pinned):**
- These app containers on .228 run as **plain `podman run --restart=unless-stopped` — NOT quadlet
  units.** `podman inspect electrumx` → `PODMAN_SYSTEMD_UNIT` is EMPTY; `systemctl --user stop
  electrumx.service` → `Unit electrumx.service not loaded (rc=5)`. The only `.container` quadlet
  files on disk are the 4 UI companions + `home-assistant`; **`bitcoin-core.container` is renamed
  `.disabled-20260506`**. ⇒ **The "Quadlet-everywhere ~96% migrated" claim in
  `app-registry-status-2026-06-21.md` is WRONG for backend apps** (the survey read a misleading
  signal). This is itself a finding the gate was meant to catch.
- `prod_orchestrator::stop()` (prod_orchestrator.rs:2890) does: `self.loaded(app_id)?` →
  `quadlet::stop_service("{name}.service")` (fails-soft for non-quadlet) → `runtime.stop_container`
  (podman fallback). For electrumx, `compute_container_name` → bare `"electrumx"` (correct target).
  filebrowser hits the podman fallback and stops; electrumx does NOT ⇒ suspect `loaded("electrumx")`
  **erroring before the fallback** (manifest not loaded in orchestrator) AND the error not being
  classed as `is_unknown_app_id_error` (so `do_orchestrator_package_stop` never falls back to the
  plain-podman `do_package_stop`). **NEXT: confirm by capturing the `STOP:`/`STOP FAIL:` line**
  (the best-effort install-log was empty on .228 — promote it to a `tracing::error!` so the failure
  reason is visible in journalctl, or reproduce locally with the mock orchestrator) and inspecting
  `loaded()` + `is_unknown_app_id_error`.
- The **stop→stopped STATE reporting is correct** when the container actually stops: server.rs:1334
  keeps a `--rm`'d app visible as `Stopped` via the `user_stopped` guard (proven on filebrowser).
  So the bug is purely "container never stops", not "state not reported".

**Decisions needed before coding the fix:**
1. **Quadlet vs plain-podman — which is the intended runtime?** `bitcoin-core.container.disabled`
   (May 6) suggests the fleet *reverted* backend apps from quadlet to plain podman. If plain-podman
   is intended, `package.stop` MUST reliably reach `runtime.stop_container` for every app (fix the
   `loaded()`/fallback gap). If quadlet is intended, these apps aren't migrated and the
   status-doc/Pillar-1 claims must be corrected + the migration actually done. **Check .198
   (untouched today) for ground truth: is its `electrumx` quadlet-managed or plain podman?**
2. Whether to also widen the gate's stop-wait timeout for heavy apps (secondary; the primary bug is
   that the container never stops at all, so timeout tuning won't fix electrumx).

### MY-SESSION ERRATA (own it on resume)
- I ran the gate with `ARCHY_ALLOW_CASCADE_DESTRUCTIVE=1`, which is **NOT** the canonical gate (that
  is `ARCHY_ALLOW_DESTRUCTIVE=1` only — stop/start/restart, no uninstall/reinstall; see run-20x.sh
  "Suggested release-gate invocation"). Cascade ran uninstall/reinstall on every app and, when I
  killed the run mid-iteration, left bitcoin-knots/electrumx/btcpay/fedimint/immich uninstalled or
  stranded. **I fully restored .228** (reinstalled bitcoin-knots with the correct image
  `146.59.87.168:3000/lfg2025/bitcoin-knots:latest`; started the rest; cleared a stale
  `user-stopped.json`). Verified healthy: UI 200, 35 containers, 17 apps `running`.
- Reinstall gotcha: `package.install` needs a REAL image ref in `dockerImage`; a bare app name
  → `Invalid Docker image format`.

### NEXT STEPS (in order)
1. **Get .198 ground truth on quadlet** (one SSH at a time — .198 sshd wedges under concurrent
   sessions). `podman inspect electrumx --format '{{index .Config.Labels "PODMAN_SYSTEMD_UNIT"}}'`
   and `ls ~/.config/containers/systemd/`. Decides the quadlet-vs-podman question above.
2. **Fix the `package.stop` propagation bug** so every app's stop reaches `runtime.stop_container`
   (the podman fallback) when no quadlet unit exists — i.e. `loaded()` failure / non-unknown-app
   error must not abort the stop. Add a unit/integration test (mock orchestrator) reproducing
   electrumx-style "no quadlet unit" stop.
3. **Re-run the CANONICAL gate** `ARCHY_ITERATIONS=5 ARCHY_ALLOW_DESTRUCTIVE=1
   tests/lifecycle/run-20x.sh` on .228, then .198. (Do NOT set CASCADE unless deliberately testing
   uninstall/reinstall, and never kill it mid-iteration on a real node.)
4. **netbird migration (#20 phase 4)** — same pattern; assess setup steps first (TLS cert gen,
   config files, resolver IP — may need host-file-write hooks beyond exec/copy_from_host; legacy is
   install_netbird_stack in stacks.rs).
5. Then single-container legacy apps onto the orchestrator install flow; then demote the banner.

### KNOWN ISSUES / WATCH-OUTS
- **.198 is a weak/loaded node** (load avg ~3–5). The generic reconcile recreates
  containers it deems unhealthy; under load, false-failing health checks → churn. The
  tcp-health fix (`e2a012d0`) mitigated the frontend case. If the lifecycle gate churns on
  .198, look for other apps whose http health checks false-fail under load → prefer tcp.
- **Many concurrent SSH sessions to .198 wedge its sshd** (MaxStartups) — it pings but SSH
  hangs for minutes. Use ONE ssh at a time to .198; `pkill -f 192.168.1.198` to clear strays.
- Hook `exec` only works in the scoped form (committed). `copy_from_host` is direct `cp`.

### DEPLOY / VERIFY FACTS (both nodes, ISO Debian, glibc 2.41 — binary built on .116 runs on both)
- **Build:** `cd core && CARGO_INCREMENTAL=0 cargo build --release -p archipelago`
  (~12 min, opt-level=3). Binary at `core/target/release/archipelago`. Linker
  "undefined hidden symbol" → rebuild with CARGO_INCREMENTAL=0. `archipelago` is a
  bin-only crate (no lib). Filtered tests: `cargo test -p archipelago --bin archipelago -- hooks quadlet`.
- **Sideload:** `scp binary $H:/tmp/archipelago-new` → `sudo systemctl stop archipelago;
  sudo cp /tmp/archipelago-new /usr/local/bin/archipelago; sudo chmod +x …; sudo systemctl
  start archipelago`. Containers SURVIVE the restart (--restart unless-stopped +
  podman-restart.service). Binary path is /usr/local/bin/archipelago.
- **Manifests** live at /opt/archipelago/apps/<app_id>/manifest.yml (root-owned ok). The
  orchestrator CACHES them at startup → **edit on disk then RESTART archipelago to reload**.
  Bulk deploy: `tar czf t.tgz -C apps indeedhub indeedhub-postgres indeedhub-redis
  indeedhub-minio indeedhub-relay indeedhub-api indeedhub-ffmpeg`; scp; `sudo tar xzf t.tgz
  -C /opt/archipelago/apps`.
- **Nodes:** .228 = 192.168.1.228, SSH pw `archipelago`, RPC/UI pw `password123` (https).
  .198 = 192.168.1.198, SSH pw `archipelago`, **RPC/UI pw `ThisIsWeb54321@`** (https). Both
  have the 7-container indeedhub stack + secrets + named volumes pre-existing.
- **Trigger install via RPC:** `auth.login` (sets session+csrf cookies) → send the csrf
  cookie value as `X-CSRF-Token` header → `package.install` with params
  `{"id":"indeedhub","dockerImage":"<any>"}` (dockerImage required even for stacks; install
  is async → returns `{"status":"installing"}`). install logs go to
  /var/log/archipelago/container-installs.log (best-effort) AND journalctl -u archipelago.
- **Fresh-create test recipe:** `podman rm -f indeedhub` (stateless frontend) → package.install
  indeedhub → expect install_fresh + post_install hook (all 4 steps `ok`) + UI 200 on :7778
  (/ , /nostr-provider.js, /api/). On adoption the frontend is NoOp (hook does NOT run —
  install_fresh is the only hook trigger).

## 9. Documentation map (what survives)

This master plan is the hub. Authoritative standalone docs (linked above), kept:

- **Design:** `architecture.md`, `app-developer-guide.md`,
  `APP-PACKAGING-MIGRATION-PLAN.md`, `registry-manifest-design.md`,
  `marketplace-protocol.md`, `dht-distribution-design.md`,
  `multi-node-architecture.md`, `rust-orchestrator-migration.md`,
  `bulletproof-containers.md`, `three-mode-ui-design.md`, `dual-ecash-design.md`,
  `meshroller-integration-design.md`, `phase4-streaming-ecash-plan.md`, `adr/*`.
- **Reference:** `app-manifest-spec.md`, `api-reference.md`, `developer-guide.md`,
  `operations-runbook.md`, `troubleshooting.md`, `user-walkthrough.md`,
  `bitcoin-rpc-relay.md`, `security-code-audit-2026-03.md`, `GAMEPAD-NAV.md`,
  `SEED-VERIFICATION.md`, `hotfix-process.md`, `app-registry-status-2026-06-21.md`.

All dated handoffs/resumes/transcripts/superseded trackers were consolidated here
and removed (recoverable via git) on 2026-06-21.