archy/docs/PRODUCTION-MASTER-PLAN.md

# 🚩 PRODUCTION MASTER PLAN — Archipelago App Platform & Registry

> **THIS IS THE AUTHORITATIVE PLAN. Agents: read this first and keep it open until
> the production test gate (§5) is green.** It overrides ad-hoc direction and
> supersedes all prior roadmap/handoff/status docs. When the gate passes, remove
> the priority banner and demote this doc.
>
> Last updated: 2026-06-21 · Binary: v1.7.99-alpha

---

## 1. The North Star

Make Archipelago a **world-class, developer-ready app platform** where:

1. **Every app is manifest-driven** — install/run/update/uninstall needs only the
   app's manifest (+ catalog entry). **Zero OS-level code reliance**: no per-app
   Rust installers, no `sudo mkdir/chown`, no host provisioning.
2. **Manifests are distributed via the (signed) registry**, not baked into the
   binary OTA as disk files. Bumping/adding an app = a signed catalog change.
3. **Third-party developers can build and ship apps via an external registry** —
   a decentralized marketplace (DID-signed manifests, Nostr discovery, reputation),
   not a gatekept central store. `archy app validate/render/install/test` tooling.
4. The platform stays **rootless, secure-by-default, elegant, robust, and
   100%-uptime-capable** (reboot-survivable, self-healing, no data loss on migrate).

**Definition of done:** the production test gate (§5) is green for the app set on
real nodes. Until then, this plan is the priority.

## 2. Invariants (never violate)

- **Rootless Podman only.** No rootful, no Docker-socket mounts, no privileged
  containers unless explicitly approved. (ADR-001, ADR-009.)
- **No app-specific business logic in the Rust backend.** The orchestrator owns
  the lifecycle state machine; apps are declarative. Legacy `install_immich_stack`
  (hardcoded `podman run` + `sudo chown`) is the anti-pattern being deleted.
- **Secrets are manifest-declared** (`generated_secrets`, materialised by
  `container::secrets` 0600/rootless, idempotent + self-healing) — never hardcoded,
  per-app, or logged. Replaces the deleted `ensure_fmcd_password`.
- **Migrations never destroy data.** Preserve `/var/lib/archipelago/<app>`,
  generated secrets, displayed credentials, public ports, and adoption container
  names. Always provide a rollback path. Stop/recreate only when necessary.
- **Verify on a real node (.228, then .198) before any tag.**

## 3. Current state (2026-06-21)

- **~40 apps are manifest-based and Quadlet-migrated** (survive
  `archipelago.service` restart + reboot). Exhaustive per-app table:
  `docs/app-registry-status-2026-06-21.md`.
- **Legacy holdout: immich** — the one app with **no manifest** and a hardcoded
  Rust stack installer (in-cgroup, not Quadlet). 3 containers, healthy, live data.
  The migration proof case.
- **Manifests still travel by OTA disk rsync** (`apps/ → /opt/archipelago/apps`).
  The signed catalog (`app-catalog.json`) currently distributes **only image
  overrides** — not full manifests. Gap closed by workstream B.
- **The 4 companions** (`archy-bitcoin-ui`, `-lnd-ui`, `-electrs-ui`,
  `-fedimint-ui`) build from `docker/<name>` contexts via `companion.rs`, not the
  manifest registry — a later phase folds them in.
- **No app has passed the formal production gate (5× for now, was 20×).** That is the blocker.

## 4. Workstreams (each links its authoritative detail doc)

| # | Workstream | Detail doc | Status |
|---|-----------|-----------|--------|
| A | **Manifest-driven app platform** — packaging contract, single/multi-container runtime, routing, controlled hooks, dev tooling (6 phases, security model, migration rules) | `APP-PACKAGING-MIGRATION-PLAN.md` | mostly done; immich + multi-container polish remain |
| B | **Registry-distributed manifests** — catalog carries full signed manifest; orchestrator installs from registry; disk = migration fallback | `registry-manifest-design.md` | **phases 1+2 done** (node consume + opt-in publisher embed); not yet flipped on for the fleet |
| C | **Developer-ready external registry** — 3rd-party DID-signed manifests, decentralized Nostr discovery (NIP-78 kind 30078) + trust score, `archy app …` tooling | `marketplace-protocol.md`, `app-developer-guide.md` | design exists; tooling + trust UX pending |
| D | **Distribution backbone** — signed catalog, BLAKE3 content-addressing, iroh swarm (origin-always-wins) | `dht-distribution-design.md` | phases 0–2 code-complete (worktree) |
| E | **Production test gate** — 5× lifecycle on .228 + .198 (for now; was 20×), per-app L1/L2 matrix | `tests/lifecycle/TESTING.md`, `bulletproof-containers.md` | **never green — exit criterion** |

**Orchestrator architecture** (foundation for A/B): `rust-orchestrator-migration.md`
(ProdContainerOrchestrator, BootReconciler 30s level-triggered reconcile, adoption
scan, Quadlet rendering) and `bulletproof-containers.md` (the six container failure
modes FM1–FM6 + the desired-state-first reconciler that fixes them).

## 5. Production test gate (exit criterion)

An app is **production-ready** only when `tests/lifecycle/run-20x.sh` is green
across the full matrix — install / UI-reachable / stop / start / restart /
reinstall / **reboot-survive** / **archipelago-restart-survive** / uninstall —
**5× on .228 AND .198 for now** (`ARCHY_ITERATIONS=5`; temporarily reduced from
20× — restore to 20× before the final ship). All 8 gate checkboxes in `tests/lifecycle/TESTING.md`
are currently unchecked. Coverage today: L0 unit (631 ●), L1 RPC ● for 6 core apps,
L2 UI ● dashboard + proxies; L3 survival ◐; ~30 apps have zero automated coverage.

## 6. Immediate sequence (live workstream)

1. ✅ **B-phase 1** — `manifest` field on `AppCatalogEntry`; `load_manifests`
   catalog-wins merge; `manifest_dir` kept (build-source catalog manifests skipped
   in phase 1); unit tests. *(commit 220666d3)*
2. ✅ **B-phase 2** — `EMBED_MANIFESTS` publisher generator + round-trip guard.
   *(7bfbe8fe; signing via existing ceremony — not yet flipped on for the fleet.)*
3. ✅ **C immich proof** — immich is a manifest-driven stack (immich + immich-postgres
   + immich-redis) installed via `install_stack_via_orchestrator`; legacy installer
   is now fallback-only. Live-migrated + verified on .228. Found+fixed: container_name
   duplicate-on-shared-PGDATA, version-digit validation, partial-fallback hardening,
   data_uid 100998. Canonical app_id `immich` (title+icon). *(9e6c5370, d5ef4573)*
4. ✅ **Reboot-survival** — podman-restart.service enabled (startup, fleet-wide)
   for the podman-`--restart` path. *(f160e0c4)*
5. ◻ **Verify on .198** (immich migration validated on .228 only so far).
6. ◻ **E** — run the 5× gate (`ARCHY_ITERATIONS=5`, was 20×); fix until green.
7. ◻ Demote this banner.

**Not yet done / deliberate follow-ups:** flip `EMBED_MANIFESTS` on for the
published catalog (then sign) to actually distribute manifests via the registry;
Phase-3 `use_quadlet_backends` rollout so orchestrator backends are Quadlet (not
just podman-`--restart`); immich on .198.

## 7. Release blockers & operational gotchas (durable)

Carried forward from prior handoffs (deduped against persistent memory):

- **Rootless control-plane responsiveness** — slow `podman ps`/store cleanup at
  startup must not surface a false "no apps installed" UI. **My Apps must preserve
  last-known apps during scanner backoff**, never show empty during a transient.
- **Reboot survival** — gate on ≥3 (prefer 5) consecutive clean post-reboot
  lifecycle passes. Quadlet units under `user.slice` survive `archipelago.service`
  restart; legacy in-cgroup containers get SIGKILLed and reconciled back.
- **Startup patterns** — wait on a socket/health, never `sleep`. Tailscale waits
  for its socket; Fedimint Guardian waits for Bitcoin RPC `initialblockdownload:false`
  before launching fedimintd (proxy/wait companion on :8175 during IBD).
- **Bitcoin must run full** (`txindex=1`, non-pruned) for ElectrumX/mempool.
- **Adoption** — match existing containers by name and adopt without recreate;
  record a migration version in app state; preserve Nostr signer bridges
  (IndeeHub needs `/nostr-provider.js` served, not just port reachability).
- **Image presence** — use bounded targeted `podman image inspect`, not
  `podman image exists` (avoids store-walk stalls).
- **Companion rebuilds** — `companion.rs` must rebuild `:latest` when the build
  context changes (staleness check), else baked-in fixes (e.g. guardian CSS) never
  reach nodes. `:local` is a manual override, never auto-rebuilt.

## 8. Roadmap

**Pipeline:** Feature Testing (internal) → User Testing (controlled hardware) →
Beta Live (public). Hardening priorities feeding the gate:

- **P0** Container app reliability — bulletproof install/health/restart/uninstall
  across all apps, dependency chains, multi-container stacks.
- **P0** Networking stack first-install → reboot-proof (WireGuard/NetBird, Tor
  hidden services, LND Connect).
- **P1** LUKS2 full-partition encryption for `/var/lib/archipelago/`
  (AES-256-XTS, Argon2id, key from setup password + hardware salt).
- **P1** Meshtastic plug-and-play parity with MeshCore.

**Post-beta (deferred — do not start until gate is green):** P2P encrypted
voice/video (WebRTC over federation via Tor); watch-only wallet + mesh BTC
hardening; paid swarm streaming + IndeeHub source (`phase4-streaming-ecash-plan.md`);
Meshroller Rust-native mesh AI (`meshroller-integration-design.md`); dual-ecash
phases 2–6 (`dual-ecash-design.md`).

## 8b. SESSION STATE + RESUME (2026-06-21, live)

**Landed + committed on main this session (newest first):**
- **#20 phase 3 — ADOPTION PATH LIVE-VERIFIED on .228 (2026-06-21).** Built
  v1.7.99-alpha, sideloaded binary + 7 manifests, restarted (stop/replace/start —
  containers survived via --restart unless-stopped + podman-restart.service). RPC
  `package.install indeedhub` → `complete`, orchestrator-first path adopted all 7
  members (`reconcile action app_id=indeedhub-* action=NoOp`), containers stayed
  **Up 4 days (NOT recreated)** — zero data/credential disruption. UI green:
  frontend :7778 → 200, nostr-provider.js → 200, **/api/ → 200 (proves
  network_aliases: frontend nginx `http://api:4000` resolved on indeedhub-net)**.
  Fleet healthy (36 containers, none down).
  **FRESH-CREATE PATH = FIXED + VERIFIED (2026-06-21).** Deleted the legacy
  indeedhub orchestrator special-cases (`b73084db`, −382 lines: reconcile_indeedhub_stack,
  start_indeedhub_backends, the 120s dependency-DNS gate, patch_indeedhub_nostr_provider,
  etc.) so "indeedhub" flows through the generic install_fresh path. Then two live fixes
  on .228: (1) frontend nginx needs `capabilities: [CHOWN,DAC_OVERRIDE,SETGID,SETUID]`
  under the orchestrator's --cap-drop=ALL (workers died "setgid(101) failed"); manifest
  fix `ff8f11b8`. (2) NOTE: manifest reload needs an archipelago restart (manifests
  cached at startup) — a disk manifest edit alone won't take. RESULT: frontend
  fresh-creates via install_fresh, caps applied, post_install hook FIRES
  (copy_from_host nostr-provider.js ✅), UI 200 (/, /nostr-provider.js, /api/).
  **KNOWN GAP (general hook capability, NOT blocking indeedhub):** the post_install
  `exec` steps fail via `podman exec` from the archipelago.service systemd cgroup
  (`crun: write cgroup.procs: Permission denied / OCI permission denied`). Harmless
  here (image bakes the nginx config so the exec steps are no-ops; copy_from_host is
  the one that matters and works). FIX = wrap the hook executor's `podman exec` in a
  transient user scope (`systemd-run --user --scope`, like `podman_user_scope`) in
  core/archipelago/src/container/hooks.rs::run_podman. Do before relying on exec hooks
  for an app whose image does NOT pre-bake its mutations.

  PRIOR (now resolved) — was: **FRESH-CREATE PATH = BLOCKED (found live 2026-06-21).** Removed the stateless
  frontend + reinstalled to exercise install_fresh → it FAILED:
  `orchestrator stack install indeedhub failed at app indeedhub: IndeedHub
  dependencies were not ready within 120s (indeedhub-api dependency DNS not ready)`,
  and the frontend was left down. Recovered manually on .228 (podman run w/ alias
  indeedhub on indeedhub-net; UI 200). ROOT CAUSE = hardcoded indeedhub orchestrator
  special-cases that predate + conflict with the manifest path:
  - prod_orchestrator `ensure_running` ~L1377: `app_id=="indeedhub"` →
    `reconcile_indeedhub_stack`, which REFUSES manifest creation when the frontend
    is absent (returns Left("stack-managed")).
  - `run_pre_start_hooks("indeedhub")` ~L2324 → `start_indeedhub_backends` →
    `wait_for_indeedhub_dependencies_ready(120)` — the gate that blocked install_fresh
    (`indeedhub_api_dependency_dns_ready` returns false while the frontend's own alias
    is absent + a getent transiently fails).
  - also `repair_indeedhub_network_aliases`, `patch_indeedhub_nostr_provider`, the
    "frontend did not stay reachable; restart" path (~L2474), `INDEEDHUB_BACKEND_*`
    consts, and a crash_recovery.rs indeedhub special-case.
  **FIX (next, its own build/deploy/test cycle):** delete these special-cases now
  that the manifest carries dependencies/network_aliases/post_install — route
  "indeedhub" through the GENERIC install_fresh + reconcile path so the frontend
  fresh-creates normally (hook fires). Then re-run the destructive lifecycle on .228
  (frontend recreate must succeed + run the hook), then .198, then the gate.
  NOTE: .228 currently runs v1.7.99-alpha (these special-cases still present) — the
  running stack is fine (adoption NoOp); only a frontend-absent event re-triggers the
  bug, and the frontend is up.
- `b1eea8c0` indeedhub (#20) **phase 3 — CODE COMPLETE, unit-tested.** 7 manifests (apps/indeedhub-{postgres,redis,minio,relay,api,
  ffmpeg} + apps/indeedhub frontend) + install_indeedhub_stack orchestrator-first
  (immich pattern). Data-preserving by construction = ADOPTION on .228: exact live
  hyphen container names, named volumes indeedhub-*-data, dedicated indeedhub-net +
  network_aliases [postgres|redis|minio|relay|api], generated_secrets reuse live
  /var/lib/archipelago/secrets values (ensure_one no-ops on existing). Frontend
  carries the post_install nginx hook (replaces patch_indeedhub_nostr_provider;
  defensive since indeedhub:1.0.0 already bakes it). .228 GROUND TRUTH captured:
  7 containers Up, volumes indeedhub-{postgres,redis,minio,relay}-data, network
  indeedhub-net; frontend nginx upstreams api:4000/minio:9000/relay:8080; image
  already bakes X-Frame strip + nostr-provider.js (6347B) + sub_filter.
  **NEXT = live verify on .228:** build+sideload binary, restart, package.install
  indeedhub → expect adoption (NoOp, no data touch), then full lifecycle. Risk:
  service restart SIGKILL-cascade if Quadlet not fully shipped on .228.
- `b94b61f6` `network_aliases` manifest field (ContainerConfig) + podman_client &
  quadlet rendering + DNS-label validation; also fixed 4 pre-existing from_manifest
  test failures (network_policy: archy-net invalid; bind sources outside
  /var/lib/archipelago). Enables indeedhub's short aliases on indeedhub-net.
- `955c54b7` hook capability (#20) **phase 2** — `container::hooks::run_post_install`
  executor (podman exec + copy_from_host w/ allowlist canonicalise + symlink-escape
  prefix check; best-effort/idempotent) wired into `install_fresh` after container
  is up (fresh-container-only). 5 unit tests; `cargo test -p archipelago` green.
- `4c1a4e59` hook capability (#20) **phase 1** — `LifecycleHooks`/`HookStep`/`HostCopy`
  schema + validate() + re-exports + 3 schema tests; also fixed 3 pre-existing
  `ContainerConfig` test literals missing `generated_secrets` (container crate now
  compiles; `cargo test -p archipelago-container` green, 53 pass).
- `f0c6b79d` immich containers named underscore (immich_server/_postgres/_redis) to
  match runtime lifecycle code — fixes package.stop/start/restart. **immich fully
  migrated + verified on .228** (manifest-driven stack via orchestrator).
- `b0b54a96` immich lifecycle bats suite (tests/lifecycle/bats/immich.bats).
- `d5ef4573`/`9e6c5370`/`011081d1` immich migration (rename→immich, orchestrator-first).
- `f160e0c4` podman-restart.service enabled at startup (reboot-survival).
- `0860dfac` Services-tab UI (backends→Services, parent icons, categories sub-nav, swipe).
- `220666d3`/`7bfbe8fe` registry-manifest infra phases 1+2 (consume + EMBED_MANIFESTS publish).
- `192238cb` docs consolidation 56→28 + CLAUDE.md.
- `03a4ee1b` generated-secrets system + companion/quadlet fixes.

**DONE — hook capability (#20), phases 1+2 (schema + executor + wiring):**
controlled post-install hooks so indeedhub/netbird can migrate. Design:
`docs/manifest-hooks-design.md`. Schema, validate(), executor, and install-path
wiring all landed + green (commits `4c1a4e59`/`955c54b7` above). Remaining #20
phases: 3 = indeedhub migration (NEXT, below); 4 = netbird; 5 = `pre_start` hooks
(type exists, NOT yet executed — wire into `prepare_for_start` if/when needed).

**NEXT — #20 phase 3, indeedhub migration:** author 7 member manifests
(postgres/redis/minio/relay/api/ffmpeg + frontend) on archy-net with container-name
hostnames; frontend carries the `post_install` hook (strip X-Frame-Options, copy
nostr-provider.js, inject script, nginx reload — see `patch_indeedhub_nostr_provider`
in install.rs:68 for exact ops); wire `install_indeedhub_stack` orchestrator-first;
generated_secrets: indeedhub-db-password/indeedhub-jwt/indeedhub-minio-password
(reuse live values); preserve hardcoded AES_MASTER_SECRET literal + minio user
"indeeadmin". Then netbird (assess its setup steps). Then single-container legacy
apps (add to `uses_orchestrator_install_flow` allowlist in install.rs + verify each).
Then the lifecycle gate (#6) — needs harness hardening (#18) + .228 bitcoin synced.

**Test/deploy facts:** .228 = archi resilience node, UI/RPC pw `password123` (https),
SSH pw `archipelago`. Lifecycle harness runs from .116: `cd tests/lifecycle &&
ARCHY_HOST=192.168.1.228 ARCHY_SCHEME=https ARCHY_PASSWORD=password123
ARCHY_ALLOW_DESTRUCTIVE=1 ./run.sh <suite>`. RPC trigger: auth.login (sets session
+ csrf cookies) → send csrf cookie value as `X-CSRF-Token` header. package.install
needs `{"id":"<app>","dockerImage":"<any-valid-image>"}` (dockerImage required even
for stacks). Rust workspace root = `core/`. Linker `undefined hidden symbol` →
rebuild with `CARGO_INCREMENTAL=0`. immich on .228: app_id `immich`, containers
immich_server/immich_postgres/immich_redis, data dir owner 100998:100998.

## 9. Documentation map (what survives)

This master plan is the hub. Authoritative standalone docs (linked above), kept:

- **Design:** `architecture.md`, `app-developer-guide.md`,
  `APP-PACKAGING-MIGRATION-PLAN.md`, `registry-manifest-design.md`,
  `marketplace-protocol.md`, `dht-distribution-design.md`,
  `multi-node-architecture.md`, `rust-orchestrator-migration.md`,
  `bulletproof-containers.md`, `three-mode-ui-design.md`, `dual-ecash-design.md`,
  `meshroller-integration-design.md`, `phase4-streaming-ecash-plan.md`, `adr/*`.
- **Reference:** `app-manifest-spec.md`, `api-reference.md`, `developer-guide.md`,
  `operations-runbook.md`, `troubleshooting.md`, `user-walkthrough.md`,
  `bitcoin-rpc-relay.md`, `security-code-audit-2026-03.md`, `GAMEPAD-NAV.md`,
  `SEED-VERIFICATION.md`, `hotfix-process.md`, `app-registry-status-2026-06-21.md`.

All dated handoffs/resumes/transcripts/superseded trackers were consolidated here
and removed (recoverable via git) on 2026-06-21.