fix(lifecycle): resurrect fully-stopped stacks member-by-member (quadlet)

Second instance of the stop→start destruction class from tonight's gate
runs (.228): package.stop of a multi-container stack marks every member
user-stopped and quadlet-removes their containers; package.start's fallback
then resurrected only the bare package id, leaving the other members absent
AND user-stopped — indeedhub lost all 7 containers this way (mempool needed
its umbrella alias for the same reason).

- New stack_member_app_ids map (real manifest app ids, dependency order) —
  used by the start fallback when a stack has no live containers, and by
  package.restart instead of hard-failing "No containers found".
- orchestrator_uninstall_app_ids grows the missing indeedhub arm (same
  reconciler-ghost rationale as the immich arm).
- do_orchestrator_package_start no longer aborts the whole sequence when one
  member's unknown-app-id fallback fails (the #73/#74 wish) — collects and
  continues.

NOTE: extends the known §G per-app-map debt (tracker updated) — the proper
home for stack membership is a manifest field; parked post-tag by design.

Tests: package:: 48/48 incl. new fallback + indeedhub-uninstall coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

core/archipelago/src/api/rpc/package/dependencies.rs

@@ -611,12 +611,53 @@ pub(super) async fn ordered_containers_for_start(package_id: &str) -> Result<Vec
 /// error via `?`, every later member (the api + frontend) is then skipped,
 /// leaving the stack down until the health monitor recovers it minutes later.
 /// That was the source of mempool gate flakes #73 (frontend) / #74 (api).
+/// Orchestrator APP ids for a multi-container stack, in dependency-start
+/// order. Unlike `startup_order` (a union of CONTAINER-name variants across
+/// install generations, sometimes including foreign dependencies), every
+/// entry here is a real manifest app id the orchestrator can `start()` to
+/// recreate the member from scratch.
+///
+/// Used when a stack has NO live containers: under quadlet, `package.stop`
+/// removes every member container, so a subsequent `package.start` must
+/// resurrect member-by-member. Falling back to just the package id left the
+/// other members absent AND user-stopped (.228 indeedhub, 2026-07-09 — the
+/// reconciler then correctly refused to revive them).
+pub(super) fn stack_member_app_ids(package_id: &str) -> &'static [&'static str] {
+    match package_id {
+        "immich" => &["immich-postgres", "immich-redis", "immich"],
+        "indeedhub" => &[
+            "indeedhub-postgres",
+            "indeedhub-redis",
+            "indeedhub-minio",
+            "indeedhub-relay",
+            "indeedhub-api",
+            "indeedhub-ffmpeg",
+            "indeedhub",
+        ],
+        "btcpay-server" | "btcpayserver" | "btcpay" => {
+            &["archy-btcpay-db", "archy-nbxplorer", "btcpay-server"]
+        }
+        "netbird" => &["netbird-server", "netbird-dashboard", "netbird"],
+        // The legacy umbrella id maps to the split stack (the orchestrator's
+        // umbrella alias handles this too; listing it here keeps the RPC
+        // layer's fan-out explicit).
+        "mempool" | "mempool-web" => {
+            &["archy-mempool-db", "mempool-api", "archy-mempool-web"]
+        }
+        _ => &[],
+    }
+}
+
 fn order_present_containers(package_id: &str, containers: Vec<String>) -> Vec<String> {
     if containers.is_empty() {
-        // Nothing is live under any known name. Fall back to the package id so
+        // Nothing is live under any known name. For known stacks, resurrect
-        // a single-container app whose container matches its id still gets one
+        // every member via its app id (see stack_member_app_ids). Otherwise
-        // start attempt; multi-container stacks with no live members are
+        // fall back to the package id so a single-container app whose
-        // surfaced as "no containers" by the caller's emptiness check.
+        // container matches its id still gets one start attempt.
+        let members = stack_member_app_ids(package_id);
+        if !members.is_empty() {
+            return members.iter().map(|s| s.to_string()).collect();
+        }
         return vec![package_id.to_string()];
     }
     let order = startup_order(package_id);
@@ -732,10 +773,35 @@ mod tests {
     }
 
     #[test]
-    fn order_present_containers_empty_falls_back_to_package_id() {
+    fn order_present_containers_empty_resurrects_stack_members() {
+        // Under quadlet, package.stop removes every member container; the
+        // start fallback must name each member APP id so the orchestrator can
+        // recreate the whole stack (.228 indeedhub, 2026-07-09) — not just
+        // the umbrella/package id.
         assert_eq!(
             order_present_containers("mempool", vec![]),
-            vec!["mempool".to_string()]
+            vec!["archy-mempool-db", "mempool-api", "archy-mempool-web"]
+        );
+        assert_eq!(
+            order_present_containers("indeedhub", vec![]),
+            vec![
+                "indeedhub-postgres",
+                "indeedhub-redis",
+                "indeedhub-minio",
+                "indeedhub-relay",
+                "indeedhub-api",
+                "indeedhub-ffmpeg",
+                "indeedhub",
+            ]
+        );
+        assert_eq!(
+            order_present_containers("immich", vec![]),
+            vec!["immich-postgres", "immich-redis", "immich"]
+        );
+        // Single-container apps keep the package-id fallback.
+        assert_eq!(
+            order_present_containers("vaultwarden", vec![]),
+            vec!["vaultwarden".to_string()]
         );
     }
 

core/archipelago/src/api/rpc/package/runtime.rs

@@ -213,14 +213,21 @@ impl RpcHandler {
 
         let single_orchestrator_app =
             self.orchestrator.is_some() && uses_single_orchestrator_app(package_id);
-        let containers = if single_orchestrator_app {
+        let mut containers = if single_orchestrator_app {
             vec![orchestrator_app_id(package_id).to_string()]
         } else {
             get_containers_for_app(package_id).await?
         };
         if containers.is_empty() {
-            tracing::warn!("package.restart {}: no containers found", package_id);
+            // A stack whose containers were all removed (quadlet stop) can
-            return Err(anyhow::anyhow!("No containers found for {}", package_id));
+            // still be restarted: resurrect it member-by-member, same as
+            // package.start's no-live-containers fallback.
+            let members = super::dependencies::stack_member_app_ids(package_id);
+            if members.is_empty() {
+                tracing::warn!("package.restart {}: no containers found", package_id);
+                return Err(anyhow::anyhow!("No containers found for {}", package_id));
+            }
+            containers = members.iter().map(|s| s.to_string()).collect();
         }
 
         // Restart does not mark user-stopped; user wants the app to keep
@@ -765,7 +772,13 @@ async fn do_orchestrator_package_start(
         match orchestrator.start(name).await {
             Ok(()) => wait_after_orchestrator_start(name).await,
             Err(e) if is_unknown_app_id_error(&e) => {
-                do_package_start(&[name.clone()]).await?;
+                // Collect instead of `?`: aborting here skipped every later
+                // stack member (mempool gate flakes #73/#74 — see
+                // order_present_containers).
+                if let Err(e) = do_package_start(&[name.clone()]).await {
+                    tracing::error!(container = %name, error = %e, "fallback container start failed");
+                    errors.push(format!("{}: {:#}", name, e));
+                }
             }
             Err(e) => {
                 tracing::error!(container = %name, error = %e, "orchestrator start failed");
@@ -1949,6 +1962,18 @@ pub(super) fn orchestrator_uninstall_app_ids(package_id: &str) -> Vec<String> {
             "archy-btcpay-db".into(),
         ],
         "fedimint" => vec!["fedimint".into(), "fedimint-gateway".into()],
+        // IndeedHub: 7-container stack; same rationale as immich below —
+        // without this, uninstalling "indeedhub" leaves the six companions
+        // enabled for the reconciler to resurrect forever.
+        "indeedhub" => vec![
+            "indeedhub-postgres".into(),
+            "indeedhub-redis".into(),
+            "indeedhub-minio".into(),
+            "indeedhub-relay".into(),
+            "indeedhub-api".into(),
+            "indeedhub-ffmpeg".into(),
+            "indeedhub".into(),
+        ],
         // Immich: multi-container stack, mirrors `immich_stack_app_ids` in
         // stacks.rs. Without this, uninstalling "immich" only disabled the
         // orchestrator-tracked "immich" app_id — "immich-postgres" and
@@ -1996,6 +2021,25 @@ mod tests {
         assert_eq!(runtime_host_ports("gitea"), vec![3001, 2222, 3000]);
     }
 
+    #[test]
+    fn indeedhub_uninstall_covers_every_sibling_orchestrator_app_id() {
+        let ids = orchestrator_uninstall_app_ids("indeedhub");
+        for expected in [
+            "indeedhub-postgres",
+            "indeedhub-redis",
+            "indeedhub-minio",
+            "indeedhub-relay",
+            "indeedhub-api",
+            "indeedhub-ffmpeg",
+            "indeedhub",
+        ] {
+            assert!(
+                ids.iter().any(|id| id == expected),
+                "missing {expected} in {ids:?}"
+            );
+        }
+    }
+
     #[test]
     fn immich_uninstall_covers_every_sibling_orchestrator_app_id() {
         // Regression: uninstalling "immich" used to only disable the

docs/1.8.0-RELEASE-HARDENING-PLAN.md

@@ -306,7 +306,9 @@ media (latest artifact only one minor behind).
       `prod_orchestrator.rs:54-83` baseline/restart-sensitive lists). Move `baseline`,
       `restart_sensitive`, `stack_members`, `multi_container` into the manifest schema; collapse
       the five near-identical `install_*_stack()` wrappers into one generic call. **Biggest
-      maintainability win.**
+      maintainability win.** (Grew again 2026-07-09: `stack_member_app_ids` in
+      `package/dependencies.rs` — the quadlet stack-resurrection fallback — is a fifth
+      per-app map that must fold into the same manifest field.)
 - [ ] 🟢 **Route all podman/systemctl through `podman_client`.** 113 raw `Command::new("podman")`
       + 32 `systemctl` calls bypass the existing 952-LOC wrapper → untestable + the blocking-call
       risk (§C). Consolidating also unlocks unit tests for the thinly-tested `package/` handlers