test(lifecycle): add os-audit OS-wide health gate; docs: v1.7.91 resume notes

os-audit.sh: one non-destructive scorecard tying backend/RPC health, the
all-apps lifecycle audit (delegates to remote-lifecycle.sh), and the FM-guards
(port-drift, secret-completeness, orphan-container sweep, OTA-wedge). The
per-boot building block for the reboot-survival loop. FM12 check uses jq has()
not // (// treats a legit false as empty). Section A validated all-PASS on .116.

docs: v1.7.91 release-pass resume notes + the bitcoinReceive blocker writeup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## v1.7.91-alpha (2026-06-14)
+
+- Apps you've installed now reliably show their "Open" button again. Some apps — including Jellyfin, BTCPay Server, Fedimint, Gitea and Portainer — were running fine but their launch link sometimes went missing, so there was no way to open them from the home screen. They now open correctly.
+- Receiving Bitcoin is more dependable: if the wallet's internal connection details drift after a restart, it now repairs them on its own, and any error it does hit is reported clearly instead of as a generic failure or a misleading "wallet locked" message.
+- Installing Bitcoin now sets itself up correctly without manual help — a security credential that could previously be missing and stop Bitcoin from starting is created automatically before it launches.
+- The Electrum server app is back on the home screen and can be launched again.
+- Behind the scenes, the release now runs an expanded automated test suite before shipping, so these kinds of issues are caught earlier.
+
 ## v1.7.90-alpha (2026-06-13)
 
 - Generating a Bitcoin receive address works again — the wallet now requests the correct address type, fixing the "400 Bad Request" error when creating an address.

apps/electrumx/manifest.yml

@@ -51,6 +51,20 @@ app:
     - CACHE_MB=1024
     - MAX_SEND=10000000
 
+  # The ElectrumX dashboard tile is served by the host-networked companion UI
+  # (archy-electrs-ui) on port 50002, NOT by this container. Declaring it here
+  # lets the catalog generator emit electrumx -> 50002 into GENERATED_APP_PORTS
+  # so the tile resolves a launch URL without relying on the hand-maintained
+  # override in appSessionConfig.ts (which the generator can clobber). The
+  # backend only validates this block — it does not proxy/health-check it.
+  interfaces:
+    main:
+      name: Web UI
+      description: ElectrumX server status and connection details
+      type: ui
+      port: 50002
+      protocol: http
+
   health_check:
     type: tcp
     endpoint: localhost:50001

core/Cargo.lock

@@ -80,7 +80,7 @@ checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
 
 [[package]]
 name = "archipelago"
-version = "1.7.89-alpha"
+version = "1.7.90-alpha"
 dependencies = [
  "anyhow",
  "archipelago-container",

core/archipelago/src/api/mod.rs

@@ -1,4 +1,4 @@
 mod handler;
-mod rpc;
+pub(crate) mod rpc;
 
 pub use handler::ApiHandler;

core/archipelago/src/api/rpc/bitcoin_relay.rs

@@ -107,7 +107,7 @@ struct TrustedRelayPeer {
 }
 
 #[derive(Debug, Clone)]
-struct TxRelayCredentials {
+pub(crate) struct TxRelayCredentials {
     username: String,
     password: String,
 }
@@ -648,7 +648,13 @@ async fn txrelay_credentials_available(data_dir: &Path) -> bool {
         && fs::metadata(&client_env_path).await.is_ok()
 }
 
-async fn ensure_txrelay_credentials(data_dir: &Path) -> Result<TxRelayCredentials> {
+/// Idempotently ensure the tx-relay credential trio exists in the secrets dir:
+/// the random password, its derived `rpcauth` line, and the client env file.
+/// Bitcoin backend manifests reference `bitcoin-rpc-txrelay-rpcauth` as a
+/// required `secret_env`, so this must run before bitcoind starts — otherwise
+/// secret resolution hard-fails and the whole Bitcoin stack cascades (the .198
+/// failure). Safe to call repeatedly; it only writes what's missing or stale.
+pub(crate) async fn ensure_txrelay_credentials(data_dir: &Path) -> Result<TxRelayCredentials> {
     let (password_path, rpcauth_path, client_env_path) = txrelay_secret_paths(data_dir);
     let password = match read_trimmed(&password_path).await {
         Some(value) => value,

core/archipelago/src/api/rpc/lnd/wallet.rs

@@ -9,9 +9,15 @@ use super::LND_REST_BASE_URL;
 impl RpcHandler {
     /// Generate a new on-chain Bitcoin address.
     pub(in crate::api::rpc) async fn handle_lnd_newaddress(&self) -> Result<serde_json::Value> {
-        let (client, macaroon_hex) = self.lnd_client().await?;
+        let (client, macaroon_hex) = self.lnd_client().await.map_err(|e| {
+            tracing::warn!(error = %format!("{e:#}"), "LND newaddress: client/macaroon unavailable");
+            receive_error(
+                RECEIVE_WALLET_UNINITIALIZED,
+                "The Lightning wallet isn't set up on this node yet. Finish wallet setup, then try again.",
+            )
+        })?;
 
-        let resp = client
+        let resp = match client
             .get(format!("{LND_REST_BASE_URL}/v1/newaddress"))
             // LND's REST gateway parses `type` as the AddressType enum by its
             // proto name (or integer), NOT the lncli aliases. "p2wkh" is not a
@@ -21,13 +27,26 @@ impl RpcHandler {
             .header("Grpc-Metadata-macaroon", &macaroon_hex)
             .send()
             .await
-            .context("LND REST connection failed")?;
+        {
+            Ok(resp) => resp,
+            Err(e) => {
+                // The .116 case: LND container is up but its REST endpoint isn't
+                // reachable on the expected port (e.g. published-port drift), so
+                // the connection is refused. This is NOT a locked wallet — emit a
+                // distinct code so the UI stops mislabelling it.
+                tracing::warn!(error = %format!("{e:#}"), "LND newaddress: REST connection failed");
+                return Err(receive_error(
+                    RECEIVE_REST_UNREACHABLE,
+                    "The Lightning wallet service isn't reachable yet. It may be starting up or recovering — please try again in a moment.",
+                ));
+            }
+        };
 
         let status = resp.status();
         let raw_body = resp
             .text()
             .await
-            .context("LND address response could not be read")?;
+            .context("Bitcoin address response could not be read")?;
         let body: serde_json::Value = serde_json::from_str(&raw_body).unwrap_or_else(|_| {
             serde_json::json!({
                 "raw": raw_body,
@@ -36,11 +55,9 @@ impl RpcHandler {
 
         if !status.is_success() {
             let message = lnd_error_message(&body);
-            anyhow::bail!(
-                "Bitcoin address generation failed ({}): {}",
-                status,
-                message
-            );
+            let code = classify_lnd_address_error(&message);
+            tracing::warn!(%status, lnd_message = %message, code, "LND newaddress returned an error");
+            return Err(receive_error(code, default_receive_detail(code)));
         }
 
         if let Some(error) = body
@@ -48,14 +65,21 @@ impl RpcHandler {
             .or_else(|| body.get("message"))
             .and_then(|v| v.as_str())
         {
-            anyhow::bail!("Bitcoin address generation failed: {}", error);
+            let code = classify_lnd_address_error(error);
+            tracing::warn!(lnd_message = %error, code, "LND newaddress returned an error body");
+            return Err(receive_error(code, default_receive_detail(code)));
         }
 
         let address = body
             .get("address")
             .and_then(|v| v.as_str())
             .filter(|addr| !addr.trim().is_empty())
-            .ok_or_else(|| anyhow::anyhow!("Bitcoin address generation failed: LND did not return a Bitcoin address. The wallet may still be locked, uninitialized, or waiting for Bitcoin to sync."))?
+            .ok_or_else(|| {
+                receive_error(
+                    RECEIVE_WALLET_UNINITIALIZED,
+                    "The wallet didn't return an address yet. It may still be unlocking or waiting for Bitcoin to sync — please try again shortly.",
+                )
+            })?
             .to_string();
 
         Ok(serde_json::json!({ "address": address }))
@@ -583,9 +607,75 @@ fn lnd_error_message(body: &serde_json::Value) -> String {
         .to_string()
 }
 
+// Stable, machine-readable reason codes for receive-address failures. They are
+// embedded in the error message as a `[CODE]` token so the frontend
+// (neode-ui/src/utils/bitcoinReceive.ts) can show an accurate explanation
+// instead of guessing wallet state by substring-matching — which is what made
+// .228 report "wallet is locked" when LND's REST was merely unreachable.
+//
+// Every receive error string starts with "Bitcoin address" so it survives the
+// RPC error sanitizer (api/rpc/middleware.rs) unchanged rather than being
+// flattened to the generic "Operation failed" message (the .116 symptom).
+pub(crate) const RECEIVE_REST_UNREACHABLE: &str = "LND_REST_UNREACHABLE";
+pub(crate) const RECEIVE_WALLET_LOCKED: &str = "LND_WALLET_LOCKED";
+pub(crate) const RECEIVE_WALLET_UNINITIALIZED: &str = "LND_WALLET_UNINITIALIZED";
+pub(crate) const RECEIVE_SYNCING: &str = "LND_SYNCING";
+pub(crate) const RECEIVE_LND_ERROR: &str = "LND_ERROR";
+
+/// Build a receive-address error carrying a reason code the UI can map.
+fn receive_error(code: &str, detail: &str) -> anyhow::Error {
+    anyhow::anyhow!("Bitcoin address unavailable [{code}]: {detail}")
+}
+
+/// A sensible default human message per code (used for non-UI callers and logs;
+/// the frontend renders its own copy from the code).
+fn default_receive_detail(code: &str) -> &'static str {
+    match code {
+        RECEIVE_REST_UNREACHABLE => {
+            "The Lightning wallet service isn't reachable yet. It may be starting up or recovering — please try again in a moment."
+        }
+        RECEIVE_WALLET_LOCKED => {
+            "The Lightning wallet is locked. Unlock it (or finish wallet setup), then try again."
+        }
+        RECEIVE_WALLET_UNINITIALIZED => {
+            "The Lightning wallet isn't set up yet. Finish wallet setup, then try again."
+        }
+        RECEIVE_SYNCING => {
+            "The wallet is still syncing with the Bitcoin network. Please try again once it has caught up."
+        }
+        _ => "Couldn't generate a Bitcoin address right now. Please try again shortly.",
+    }
+}
+
+/// Classify a non-2xx LND error body/message into a reason code. The wording of
+/// LND's REST errors is stable enough to bucket: a locked wallet, an
+/// uninitialized wallet, a syncing chain, or some other failure.
+fn classify_lnd_address_error(message: &str) -> &'static str {
+    let m = message.to_lowercase();
+    if m.contains("locked") || m.contains("unlock") {
+        RECEIVE_WALLET_LOCKED
+    } else if m.contains("synchroniz")
+        || m.contains("syncing")
+        || m.contains("not yet ready")
+        || m.contains("in the process of starting")
+    {
+        RECEIVE_SYNCING
+    } else if m.contains("wallet not found")
+        || m.contains("not exist")
+        || m.contains("uninitialized")
+        || m.contains("not initialized")
+        || m.contains("create a wallet")
+        || m.contains("no wallet")
+    {
+        RECEIVE_WALLET_UNINITIALIZED
+    } else {
+        RECEIVE_LND_ERROR
+    }
+}
+
 #[cfg(test)]
 mod tests {
-    use super::lnd_error_message;
+    use super::*;
 
     #[test]
     fn lnd_error_message_prefers_message_field() {
@@ -604,4 +694,46 @@ mod tests {
             "unknown LND error"
         );
     }
+
+    #[test]
+    fn classify_locked_wallet() {
+        assert_eq!(
+            classify_lnd_address_error("wallet locked, please unlock"),
+            RECEIVE_WALLET_LOCKED
+        );
+    }
+
+    #[test]
+    fn classify_uninitialized_wallet() {
+        assert_eq!(
+            classify_lnd_address_error("wallet not found, create a wallet first"),
+            RECEIVE_WALLET_UNINITIALIZED
+        );
+    }
+
+    #[test]
+    fn classify_syncing() {
+        assert_eq!(
+            classify_lnd_address_error("server is still in the process of starting"),
+            RECEIVE_SYNCING
+        );
+    }
+
+    #[test]
+    fn classify_unknown_is_generic_error() {
+        assert_eq!(
+            classify_lnd_address_error("some other failure"),
+            RECEIVE_LND_ERROR
+        );
+    }
+
+    #[test]
+    fn receive_error_starts_with_sanitizer_safe_prefix_and_embeds_code() {
+        // Must start with "Bitcoin address" (survives the RPC error sanitizer)
+        // and carry the [CODE] token the frontend parses.
+        let err = receive_error(RECEIVE_REST_UNREACHABLE, "unreachable");
+        let s = format!("{err}");
+        assert!(s.starts_with("Bitcoin address"), "got: {s}");
+        assert!(s.contains("[LND_REST_UNREACHABLE]"), "got: {s}");
+    }
 }

core/archipelago/src/container/docker_packages.rs

@@ -699,7 +699,7 @@ async fn reachable_lan_address(app_id: &str, candidate: Option<String>) -> Optio
     if !requires_reachable_launch(app_id) {
         return Some(url);
     }
-    let Some(port) = url.rsplit(':').next().and_then(|p| p.parse::<u16>().ok()) else {
+    let Some(port) = launch_url_port(&url) else {
         return None;
     };
     if launch_port_reachable(port).await {
@@ -710,6 +710,23 @@ async fn reachable_lan_address(app_id: &str, candidate: Option<String>) -> Optio
     }
 }
 
+/// Extract the TCP port from a launch URL's authority.
+///
+/// The candidate URL can carry a path when it comes from a manifest
+/// `interfaces.main` declaration (e.g. `http://localhost:8096/`). A naive
+/// `rsplit(':')` then yields `"8096/"`, which fails to parse and silently
+/// drops a reachable launch URL. Reading digits after the final colon mirrors
+/// `port_from_url` in the RPC layer and tolerates a trailing path.
+fn launch_url_port(url: &str) -> Option<u16> {
+    let after_colon = url.rsplit_once(':')?.1;
+    after_colon
+        .chars()
+        .take_while(|c| c.is_ascii_digit())
+        .collect::<String>()
+        .parse::<u16>()
+        .ok()
+}
+
 async fn launch_port_reachable(port: u16) -> bool {
     matches!(
         tokio::time::timeout(
@@ -788,3 +805,26 @@ fn package_state_str(state: &PackageState) -> &str {
         PackageState::Updating => "updating",
     }
 }
+
+#[cfg(test)]
+mod launch_url_port_tests {
+    use super::launch_url_port;
+
+    #[test]
+    fn parses_port_with_trailing_path() {
+        // Regression: manifest interfaces.main yields a path-suffixed URL.
+        // The old rsplit(':') parse produced "8096/" and dropped the URL.
+        assert_eq!(launch_url_port("http://localhost:8096/"), Some(8096));
+        assert_eq!(launch_url_port("http://localhost:8175/admin"), Some(8175));
+    }
+
+    #[test]
+    fn parses_bare_authority_port() {
+        assert_eq!(launch_url_port("http://localhost:8083"), Some(8083));
+    }
+
+    #[test]
+    fn rejects_url_without_port() {
+        assert_eq!(launch_url_port("http://localhost/"), None);
+    }
+}

core/archipelago/src/container/prod_orchestrator.rs

@@ -297,6 +297,53 @@ async fn wait_for_manifest_host_ports(manifest: &AppManifest, timeout_secs: u64)
     Ok(())
 }
 
+/// Pure published-port drift check. `port_bindings_json` is the JSON that
+/// `podman inspect --format '{{json .HostConfig.PortBindings}}'` emits, e.g.
+/// `{"8080/tcp":[{"HostIp":"","HostPort":"18080"}]}`. Returns true only when a
+/// manifest container-port is positively published to a *different* host port
+/// than the manifest now asks for. Absence of a binding is deliberately NOT
+/// treated as drift here — that case is handled by the host-port repair/restart
+/// path and by host-networked apps that publish nothing — so we never trigger a
+/// destructive recreate on a false positive.
+fn host_port_bindings_drifted(
+    port_bindings_json: &str,
+    manifest_ports: &[archipelago_container::manifest::PortMapping],
+) -> bool {
+    let parsed: serde_json::Value = match serde_json::from_str(port_bindings_json) {
+        Ok(v) => v,
+        Err(_) => return false,
+    };
+    let Some(map) = parsed.as_object() else {
+        return false;
+    };
+    for port in manifest_ports {
+        let proto = if port.protocol.is_empty() {
+            "tcp"
+        } else {
+            port.protocol.as_str()
+        };
+        let key = format!("{}/{}", port.container, proto);
+        let Some(bindings) = map.get(&key).and_then(|b| b.as_array()) else {
+            // Container-port not currently published — not our case.
+            continue;
+        };
+        if bindings.is_empty() {
+            continue;
+        }
+        let expected = port.host.to_string();
+        let matches_expected = bindings.iter().any(|b| {
+            b.get("HostPort")
+                .and_then(|h| h.as_str())
+                .map(|h| h == expected)
+                .unwrap_or(false)
+        });
+        if !matches_expected {
+            return true;
+        }
+    }
+    false
+}
+
 async fn ensure_user_podman_socket() -> Result<()> {
     let socket_path = "/run/user/1000/podman/podman.sock";
     if podman_socket_accepts_connections(socket_path).await {
@@ -734,8 +781,19 @@ struct FileSecretsProvider {
 impl SecretsProvider for FileSecretsProvider {
     fn read(&self, name: &str) -> std::result::Result<String, ManifestError> {
         let path = self.root.join(name);
-        let data = std::fs::read_to_string(&path).map_err(ManifestError::Io)?;
-        Ok(data.trim().to_string())
+        match std::fs::read_to_string(&path) {
+            Ok(data) => Ok(data.trim().to_string()),
+            // Name the missing secret explicitly so the failure is actionable
+            // instead of a bare "IO error: No such file or directory" that hides
+            // which secret (and which app) is blocked.
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
+                Err(ManifestError::Invalid(format!(
+                    "required secret '{name}' is missing (expected at {}); the app cannot start until it is generated",
+                    path.display()
+                )))
+            }
+            Err(e) => Err(ManifestError::Io(e)),
+        }
     }
 }
 
@@ -1054,6 +1112,7 @@ impl ProdContainerOrchestrator {
         let lock = self.app_lock(&app_id).await;
         let _guard = lock.lock().await;
 
+        self.ensure_app_secrets(&app_id).await?;
         let mut resolved_manifest = lm.manifest.clone();
         self.resolve_dynamic_env(&mut resolved_manifest)?;
         let name = compute_container_name(&lm.manifest);
@@ -1094,6 +1153,22 @@ impl ProdContainerOrchestrator {
                             self.run_post_start_hooks(&app_id).await?;
                             return Ok(ReconcileAction::Started);
                         }
+                        // Published-port drift means the container exists but maps
+                        // its ports to the wrong host ports (e.g. lnd REST stuck on
+                        // host 8080 while the manifest/clients expect 18080, as seen
+                        // on .116). The container is already non-functional, so
+                        // recreate it even for restart-sensitive apps during boot —
+                        // leaving it "untouched" would perpetuate the breakage.
+                        if self
+                            .container_ports_drifted(&name, &resolved_manifest)
+                            .await
+                        {
+                            tracing::info!(app_id = %app_id, container = %name, "container published-port drift detected — recreating");
+                            let _ = self.runtime.stop_container(&name).await;
+                            let _ = self.runtime.remove_container(&name).await;
+                            self.install_fresh(lm).await?;
+                            return Ok(ReconcileAction::Installed);
+                        }
                         if self.container_env_drifted(&name, &resolved_manifest).await {
                             if mode == ReconcileMode::ExistingOnly
                                 && is_restart_sensitive_app(&app_id)
@@ -1154,8 +1229,12 @@ impl ProdContainerOrchestrator {
                         // reading it. A Rewritten outcome is fine here — we're
                         // about to start from a stopped state anyway.
                         self.prepare_for_start(&resolved_manifest).await?;
-                        if self.container_env_drifted(&name, &resolved_manifest).await {
-                            tracing::info!(app_id = %app_id, container = %name, "stopped container env drift detected — recreating");
+                        if self.container_env_drifted(&name, &resolved_manifest).await
+                            || self
+                                .container_ports_drifted(&name, &resolved_manifest)
+                                .await
+                        {
+                            tracing::info!(app_id = %app_id, container = %name, "stopped container env/port drift detected — recreating");
                             let _ = self.runtime.remove_container(&name).await;
                             self.install_fresh(lm).await?;
                             return Ok(ReconcileAction::Installed);
@@ -1297,6 +1376,7 @@ impl ProdContainerOrchestrator {
 
     /// Build-or-pull, create, start. Assumes the per-app mutex is already held.
     async fn install_fresh(&self, lm: &LoadedManifest) -> Result<()> {
+        self.ensure_app_secrets(&lm.manifest.app.id).await?;
         let mut resolved_manifest = lm.manifest.clone();
         self.resolve_dynamic_env(&mut resolved_manifest)?;
 
@@ -2300,6 +2380,25 @@ impl ProdContainerOrchestrator {
         Self::detect_disk_gb()
     }
 
+    /// Ensure app-specific secrets exist *before* env resolution. The Bitcoin
+    /// backends reference `bitcoin-rpc-txrelay-rpcauth` as a required
+    /// `secret_env`; it is normally created by the tx-relay flow, so nodes that
+    /// never used tx-relay lack it and `resolve_secret_env` hard-fails — taking
+    /// bitcoind (and everything that depends on it) down. Generating it here,
+    /// idempotently, lets reconcile/install self-heal that state (the .198 case)
+    /// instead of cascading. Must be called before every `resolve_dynamic_env`.
+    async fn ensure_app_secrets(&self, app_id: &str) -> Result<()> {
+        if cfg!(test) {
+            return Ok(());
+        }
+        if matches!(app_id, "bitcoin-knots" | "bitcoin-core" | "bitcoin") {
+            crate::api::rpc::bitcoin_relay::ensure_txrelay_credentials(&self.data_dir)
+                .await
+                .context("ensuring bitcoin tx-relay credentials")?;
+        }
+        Ok(())
+    }
+
     fn resolve_dynamic_env(&self, manifest: &mut AppManifest) -> Result<()> {
         let facts = self.detect_host_facts();
         let mut env = manifest.app.environment.clone();
@@ -2443,6 +2542,42 @@ impl ProdContainerOrchestrator {
         false
     }
 
+    /// True when a *running* container publishes a manifest container-port to a
+    /// different host port than the manifest now asks for (published-port
+    /// drift). This catches the class of failure seen on .116, where `lnd` was
+    /// created mapping host 8080 -> container 8080 but the current manifest maps
+    /// host 18080 -> container 8080, so every in-process REST client (which
+    /// connects to the manifest port) gets connection-refused forever while the
+    /// container looks "Up". `container_env_drifted` never inspects ports, and
+    /// `wait_for_manifest_host_ports` only restarts the stale container (which
+    /// republishes the wrong mapping), so without this check the drift is
+    /// self-perpetuating.
+    async fn container_ports_drifted(&self, name: &str, manifest: &AppManifest) -> bool {
+        if cfg!(test) {
+            return false;
+        }
+        if manifest.app.ports.is_empty() {
+            return false;
+        }
+        let inspect = tokio::process::Command::new("podman")
+            .args([
+                "inspect",
+                name,
+                "--format",
+                "{{json .HostConfig.PortBindings}}",
+            ])
+            .output()
+            .await;
+        let Ok(output) = inspect else {
+            return false;
+        };
+        if !output.status.success() {
+            return false;
+        }
+        let bindings = String::from_utf8_lossy(&output.stdout);
+        host_port_bindings_drifted(&bindings, &manifest.app.ports)
+    }
+
     async fn apply_data_uid(&self, manifest: &AppManifest) -> Result<()> {
         let Some(uid_gid) = manifest.app.container.data_uid.as_ref() else {
             return Ok(());
@@ -2752,6 +2887,7 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
         let lm = self.loaded(app_id).await?;
         let lock = self.app_lock(app_id).await;
         let _guard = lock.lock().await;
+        self.ensure_app_secrets(app_id).await?;
         let name = compute_container_name(&lm.manifest);
         let mut resolved_manifest = lm.manifest.clone();
         self.resolve_dynamic_env(&mut resolved_manifest)?;
@@ -2913,6 +3049,61 @@ mod tests {
     use async_trait::async_trait;
     use std::sync::Mutex as StdMutex;
 
+    fn port(host: u16, container: u16) -> archipelago_container::manifest::PortMapping {
+        archipelago_container::manifest::PortMapping {
+            host,
+            container,
+            protocol: "tcp".to_string(),
+        }
+    }
+
+    #[test]
+    fn port_drift_detected_when_host_port_differs() {
+        // The .116 case: container publishes container-port 8080 on host 8080,
+        // but the manifest now asks for host 18080.
+        let bindings = r#"{"8080/tcp":[{"HostIp":"","HostPort":"8080"}]}"#;
+        assert!(host_port_bindings_drifted(bindings, &[port(18080, 8080)]));
+    }
+
+    #[test]
+    fn no_drift_when_host_port_matches() {
+        let bindings = r#"{"8080/tcp":[{"HostIp":"0.0.0.0","HostPort":"18080"}]}"#;
+        assert!(!host_port_bindings_drifted(bindings, &[port(18080, 8080)]));
+    }
+
+    #[test]
+    fn no_drift_when_binding_absent() {
+        // Absence is handled elsewhere (host-port repair / host-networked apps);
+        // never treat it as drift to avoid a destructive recreate on a false
+        // positive.
+        assert!(!host_port_bindings_drifted("{}", &[port(18080, 8080)]));
+        assert!(!host_port_bindings_drifted("null", &[port(18080, 8080)]));
+    }
+
+    #[test]
+    fn no_drift_on_unparseable_bindings() {
+        assert!(!host_port_bindings_drifted(
+            "not json",
+            &[port(18080, 8080)]
+        ));
+    }
+
+    #[test]
+    fn missing_secret_error_names_the_secret() {
+        use archipelago_container::manifest::SecretsProvider;
+        let provider = FileSecretsProvider {
+            root: PathBuf::from("/nonexistent-secrets-dir-xyz"),
+        };
+        let err = provider
+            .read("bitcoin-rpc-txrelay-rpcauth")
+            .expect_err("missing secret must error");
+        let msg = err.to_string();
+        assert!(
+            msg.contains("bitcoin-rpc-txrelay-rpcauth"),
+            "error should name the missing secret, got: {msg}"
+        );
+    }
+
     /// Instrumented in-memory runtime. Every call is recorded so tests can assert
     /// the exact sequence of side effects.
     #[derive(Default)]

docs/MIGRATION_STATUS_REPORT.md

@@ -1,6 +1,153 @@
 # Migration Status Report
 
-Last updated: 2026-06-11
+Last updated: 2026-06-14
+
+## RESUME CHECKPOINT (2026-06-14, after SSH drop)
+
+State right now, so any disconnect resumes cleanly:
+
+- **`main` = `a483fe4b`** = the other agent's 4 fixes (`0ed892a4`: wallet receive / bitcoin
+  install self-heal / ElectrumX tile / extended test gate) + **my F1 fix committed on top**
+  (`launch_url_port` in `docker_packages.rs` + 3 regression tests). Tree is clean (only two
+  untracked `docs/*.md` tracking files remain). Not pushed.
+- The old isolated `archy-f1` worktree was **removed** — built the combined tree in-place.
+- ✅ **DONE — combined backend release build** (`cd core && TMPDIR=/home/archipelago/.buildtmp
+  cargo build --release -p archipelago`, 7m46s, exit 0). `/tmp` is a full tmpfs so `TMPDIR`
+  MUST point at `/home/archipelago/.buildtmp`.
+- ✅ **DONE — sideloaded + restarted on `.116`.** Backed up old binary to
+  `/usr/local/bin/archipelago.pre-f1.bak`, `install`ed new binary (root:root 755),
+  `sudo systemctl restart archipelago` (new MainPID 2885863).
+- ✅ **F1 VALIDATED LIVE on `.116` (2026-06-14).** See "FINDING F1" below — before/after proves
+  the fix. Harness focused audit `jellyfin,filebrowser` → **all checks passed, exit 0**.
+- **IMPORTANT — restart is SAFE on this node:** containers run rootless under
+  `user-1000.slice/user@1000.service/app.slice`, a DIFFERENT cgroup from
+  `/system.slice/archipelago.service`. They survived both the 01:47 and this restart
+  (bitcoin/lnd/btcpay/immich/indeedhub all intact, count stayed 36). The
+  `feedback_no_systemctl_deploy_until_quadlet` cgroup-cascade warning does NOT apply to `.116`'s
+  current config. (The reconciler does recreate a few app containers like jellyfin/fedimint on
+  adoption — normal level-triggered behavior, not casualties.)
+- **RELEASE IN PROGRESS — v1.7.91-alpha (user approved 2026-06-14).** Bundles the other agent's
+  4 fixes (`0ed892a4`) + F1 (`a483fe4b`) + changelog (`ab858271`). Steps:
+  1. ✅ Freed `/tmp` (removed stale published frontend tarballs 1.7.83→1.7.89; ~1.1G free) —
+     `create-release.sh` writes the 184MB frontend tarball to `/tmp` (hardcoded, NOT TMPDIR).
+  2. ✅ `cargo fmt -p archipelago --check` clean; curated layman changelog added + committed.
+  3. 🔄 `TMPDIR=/home/archipelago/.buildtmp scripts/create-release.sh 1.7.91-alpha`
+     (runs `tests/release/run.sh` gate → bumps Cargo.toml/package.json → builds backend+frontend
+     → manifest → commit "chore: release v1.7.91-alpha" → tag `v1.7.91-alpha`). MUST set TMPDIR
+     or cargo's ring C-build fails on the full `/tmp` tmpfs.
+  - **AFTER create-release.sh:** `scripts/publish-release-assets.sh 1.7.91-alpha gitea-vps2`
+     → `git push origin main && git push gitea-local main` → `git push --tags` (origin+gitea-local).
+     Ship target per memory: vps2 (146.59.87.168) is PRIMARY OTA manifest; tx1138 RETIRED.
+  - Verify packaged tarball actually contains the new version string before trusting the build
+    (npm run build can silently produce stale dist — see `feedback_frontend_build_verify`).
+
+## Validation node (ACTIVE)
+
+As of 2026-06-14 the app-migration lifecycle validation moves from `.198` (remote, OVH) to
+**`.116` — the local dev node (`archi-thinkpad`, `192.168.1.116`)** because it is the machine
+this session runs on, so the harness drives it over loopback instead of SSH (much faster, no
+network latency). A separate agent owns OS-level fixes + its own test harness; this track owns
+the **app-packaging migration** lifecycle validation only.
+
+How to drive the harness against `.116` (local):
+
+```bash
+ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http ARCHY_PASSWORD='ThisIsWeb54321@' \
+  ARCHY_APPS='meshtastic,jellyfin,filebrowser,uptime-kuma' \
+  tests/lifecycle/remote-lifecycle.sh        # focused, audit-only (non-destructive)
+```
+
+- `.116` serves nginx on **:80 only** (443 is tailscale's) → use `ARCHY_SCHEME=http`, `ARCHY_HOST=127.0.0.1`.
+- Local node is healthy: `update_state.json.current_version == 1.7.90-alpha`, `update_in_progress=false`
+  (the OTA self-heal that was a follow-up gap in PROGRESS_MEMORY is now confirmed resolved on .116).
+- Login password for `.116`: `ThisIsWeb54321@` (verified against `auth.login`). Note: auth.login
+  has a login rate-limiter — avoid rapid repeated attempts.
+- `.198` results below remain the prior baseline; new results are tagged `[.116]`.
+
+### [.116] audit log (newest first)
+
+- **2026-06-14 — focused audit `meshtastic,jellyfin,filebrowser,uptime-kuma` (audit-only, non-destructive):**
+  harness exit 1, FAILED checks: 1.
+  - `filebrowser` — running, pass (also passed a standalone single-app smoke run).
+  - `uptime-kuma` — running, pass.
+  - `meshtastic` — `state=absent`. Not installed on `.116` (was installed/validated on `.198`).
+    Not a regression; just node state. To exercise meshtastic here, install it first (it needs
+    `/dev/ttyUSB0`, which `.116` may not have) or drop it from the focused set on this node.
+  - `jellyfin` — **running but FAILED: "launch metadata missing: jellyfin has no lan_address".**
+    **ROOT-CAUSED 2026-06-14 — real, current bug in the working tree (a regression).** See
+    "FINDING F1" below.
+
+### [.116] FINDING F1 — manifest launch URLs with a path are silently dropped (OPEN, fix pending)
+
+**Symptom:** `jellyfin` is `running` and genuinely serving (`curl 127.0.0.1:8096/` → 302), but
+`container-list` reports `lan_address: null`, so the UI/harness sees no launch URL.
+
+**Root cause:** `core/archipelago/src/container/docker_packages.rs::reachable_lan_address()` parses
+the port out of the candidate URL with `url.rsplit(':').next()`. When the candidate comes from the
+manifest `interfaces.main` (via `PodmanClient::lan_address_for` →
+`core/container/src/podman_client.rs::manifest_primary_interface_url`), the URL **includes the
+manifest `path`** — e.g. jellyfin → `http://localhost:8096/`. Then `rsplit(':').next()` yields
+`"8096/"`, which **fails to `parse::<u16>()`**, so the function hits its `else { return None }`
+branch and drops a perfectly reachable launch URL. (Diagnostic tell: the dropped-at-parse path
+emits **no** log, whereas a genuine unreachable port logs "suppressing unreachable launch URL".
+jellyfin has no such log; uptime-kuma — whose candidate `…:3002` has no path — does.)
+
+**Why it's a regression:** the old `extract_lan_address(ports)` produced `http://localhost:PORT`
+(no path), which parsed fine. The newer manifest-interface feature appends the declared `path`,
+so any app routed through `lan_address_for` now yields `…:PORT/` and trips the parser.
+
+**Blast radius (apps in `requires_reachable_launch` whose `interfaces.main.path` = `/`):**
+`botfights`, `btcpay-server`, `fedimint`, `jellyfin`, `gitea`, `nextcloud`, `portainer`.
+(`filebrowser`/`nextcloud`/`nginx-proxy-manager`/`vaultwarden` are in `uses_allocated_launch_port`
+so they hit `extract_lan_address` first and dodge it; `grafana`/`mempool`/`uptime-kuma`/`searxng`
+have no manifest `interfaces.main` path.) On `.198` this likely went unnoticed because those apps
+weren't all running during the launch-metadata assertion, or predated the interfaces.main addition.
+
+**Fix (IMPLEMENTED in working tree, uncommitted):**
+`docker_packages.rs::reachable_lan_address` now parses the port via a new `launch_url_port()`
+helper that reads digits after the final colon (`take_while(is_ascii_digit)`), mirroring the
+RPC-layer `port_from_url`, so `http://localhost:8096/` → `Some(8096)`. Added unit tests
+(`launch_url_port_tests`) covering the trailing-path regression, the bare-authority case, and a
+no-port reject. The existing `lan_address_prefers_manifest_main_interface` test only exercised
+`lan_address_for` (which always returned `…:8175/`) and never the `reachable_lan_address` wrapper,
+which is why the bug slipped through.
+
+**Unit validation: GREEN (2026-06-14).** `cargo test -p archipelago --bin archipelago launch_url_port`
+→ 3 passed / 0 failed (trailing-path, bare-authority, no-port-reject); crate compiles clean.
+
+**Coordination note (shared tree):** the repo is on branch `fix/wallet-receive-portdrift-secrets`
+at commit `bb808df8` (= the deployed 1.7.90-alpha). A parallel agent has uncommitted changes here
+(lnd `wallet.rs`, `bitcoin_relay.rs`, `prod_orchestrator.rs`, electrumx manifest, neode-ui, new
+bats). To validate F1 in isolation (and NOT deploy their in-flight work onto the live node, nor
+disturb their tree), the live-validation build is done in a detached git worktree at
+`/home/archipelago/archy-f1` = clean `bb808df8` + only the F1 `docker_packages.rs` change. Build:
+`cd /home/archipelago/archy-f1/core && TMPDIR=/home/archipelago/.buildtmp cargo build --release -p archipelago`
+(`.116`'s `/tmp` is a 7.7G tmpfs that runs 100% full → the ring crate's C compile fails with
+"No space left on device"; redirect `TMPDIR` to `/` which has ~399G). After validation the
+worktree is removed (`git worktree remove`). NOTE: sideloading replaces the OTA-managed
+`/usr/local/bin/archipelago` with a local 1.7.90-alpha+F1 build until the next OTA — back up the
+current binary first (`/usr/local/bin/archipelago.pre-f1.bak`).
+
+**Live validation status — ✅ GREEN on `.116` (2026-06-14).** Built combined tree (`a483fe4b`),
+sideloaded, restarted `archipelago.service`. Before/after on the live node (old buggy binary → new):
+
+| app | OLD lan_address | NEW lan_address |
+|---|---|---|
+| jellyfin | `None` ❌ | `http://localhost:8096/` ✅ |
+| btcpay-server | `None` ❌ | `http://localhost:23000/` ✅ |
+| fedimint | `None` ❌ | `http://localhost:8175/` ✅ |
+| gitea | `None` ❌ | `http://localhost:3001/` ✅ |
+| portainer | `None` ❌ | `http://localhost:9000/` ✅ |
+| botfights | `None` ❌ | `http://localhost:9100/` ✅ |
+| nextcloud | `:8085` ✓ | `:8085` (unchanged — allocated-port path) |
+| filebrowser | `:8083` ✓ | `:8083` (unchanged) |
+
+Harness focused audit `jellyfin,filebrowser` → **all checks passed, exit 0**. Unit tests green.
+No container casualties (all 36 survived; see RESUME CHECKPOINT for the cgroup detail).
+
+NOTE: Do NOT run the prod binary directly to "check a version" —
+`/usr/local/bin/archipelago <anyflag>` boots a whole second node instance (learned the hard way
+2026-06-14; it exited without leaving a stray, but don't repeat).
 
 ## Goal
 

docs/PROGRESS_MEMORY.md

@@ -0,0 +1,44 @@
+# Progress Memory
+
+Last updated: 2026-06-13
+
+## Current State
+
+- `v1.7.90-alpha` release is complete, tagged, pushed, uploaded, and verified on vps2.
+- Release commit: `bb808df8` (chore: release v1.7.90-alpha).
+- Feature commit:  `c800293f` (fix: bitcoin receive, AIUI pointer input, electrs self-heal, OTA timeout).
+- Gitea tag: `v1.7.90-alpha` (on origin/gitea-vps2).
+- Live OTA manifest on the update host (146.59.87.168) now resolves to `1.7.90-alpha`; both
+  artifact download URLs (binary + frontend tarball) return HTTP 200.
+- v1.7.89-alpha was already fully shipped before this session.
+
+## What shipped in v1.7.90-alpha
+
+- Bitcoin receive address generation fixed (correct address type, no more 400).
+- AIUI/app session: on-screen pointer can click + type into app content (incl. app store
+  search); "open in new tab" opens the phone browser; mobile credential modal centered.
+- Electrs self-heals from a corrupt index and shows a percent/block-height progress screen.
+- update.rs: retired tx1138 secondary mirror dropped (one-time migration); longer download
+  timeout for slow connections.
+
+## Verification
+
+- Full release harness green (8 stages): git-diff, cargo-fmt, catalog-drift, release-manifest,
+  ui-type-check, ui-unit-tests (80 files / 655 tests), cargo-check, cargo-test-weekly.
+- Freshly built binary embeds `1.7.90-alpha` (no stale 1.7.89); frontend dist rebuilt fresh
+  (new AppSession bundle); manifest sha256 + size match on-disk artifacts.
+
+## Known gaps / follow-ups
+
+- `gitea-local` (localhost:3000) push FAILS from this node — redirects to /login (auth).
+  The v1.7.88 and v1.7.89 tags were also already missing there, so this is a pre-existing
+  condition on this node, not a v1.7.90 regression. vps2 is the primary OTA mirror and is fine.
+- OTA self-update verification on THIS node (.116) not yet observed this session — the node
+  should auto-apply from the live 1.7.90-alpha manifest; confirm
+  `update_state.json.current_version == 1.7.90-alpha` after the scheduler runs.
+
+## Resume Context
+
+- If a later session resumes, continue from the next active product/release task, not this
+  finished release.
+- Broader context: docs/WEEKLY_RELEASE_TRACKER.md, docs/RESUME.md, docs/NEXT_TERMINAL_HANDOFF.md

docs/WEEKLY_RELEASE_TRACKER.md

@@ -0,0 +1,221 @@
+# Weekly Release Tracker
+
+Last updated: 2026-06-14 (session on node .116 / archi-thinkpad)
+
+---
+
+# ▶ CURRENT PASS — v1.7.91-alpha (2026-06-14)
+
+## RESUME PROMPT (paste into a fresh session, any machine)
+
+> Resume the v1.7.91-alpha release pass for the `archy` repo (on node .116 / archi-thinkpad
+> the tree is at /home/archipelago/Projects/archy; on another machine, clone/pull `main` from
+> gitea-vps2 http://146.59.87.168:3000/lfg2025/archy.git — my fix commit is pushed there).
+> Read the top section of docs/WEEKLY_RELEASE_TRACKER.md FIRST — it has the blocker, the fix
+> already made, and exact next steps. Two goals: (1) cut & PUBLISH v1.7.91-alpha, (2) finish
+> validating + integrating the new tests/lifecycle/os-audit.sh OS-wide health harness.
+> Do NOT redo: the bitcoinReceive.ts TS2538 fix (done, committed) or the os-audit jq false-trap
+> fix (done). Resume at "EXACT NEXT STEPS — v1.7.91" below. .116 login password: ThisIsWeb54321@
+> (.116 serves http on :80 → ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http).
+
+## What happened this session
+
+- `scripts/create-release.sh 1.7.91-alpha` was running; its release gate PASSED all 7 checks,
+  backend built clean (7m22s), then it **FAILED at step [4/8] frontend build** with:
+  `src/utils/bitcoinReceive.ts(23,24): error TS2538: Type 'undefined' cannot be used as an index type.`
+  Cause: `noUncheckedIndexedAccess` — `codeMatch[1]` is `string | undefined` and was used directly
+  to index `RECEIVE_CODE_MESSAGES`. **FIXED** → `const code = message.match(/\[([A-Z_]+)\]/)?.[1]`
+  then `if (code && RECEIVE_CODE_MESSAGES[code])`. `npx vue-tsc --noEmit` is now clean (exit 0).
+  The failed run aborted BEFORE bumping the manifest (still 1.7.90) or tagging (no v1.7.91 tag),
+  but it HAD already partial-bumped Cargo.toml/package.json/locks to 1.7.91 — those partial bumps
+  are reverted (create-release.sh re-owns the bump); only the genuine TS fix + harness are committed.
+- Built a new OS-wide health harness `tests/lifecycle/os-audit.sh` (non-destructive, one scorecard):
+  Section A backend/RPC health, Section B all-apps lifecycle audit (delegates to remote-lifecycle.sh),
+  Section C FM-guards (port-drift + secret-completeness bats, orphan-container sweep). Section A
+  validated all-PASS on .116. Fixed a jq bug in the FM12 OTA-wedge check: `//` treats a legit
+  `false` as empty and fell through to "unknown" — now uses `has()`. Section B is slow (~3 min) and
+  opaque while running because output is captured (`out=$(...)`) not streamed — minor wart, TODO.
+
+## EXACT NEXT STEPS — v1.7.91 (in order)
+
+1. Confirm clean tree + on main (`git status`; create-release.sh requires `git diff --quiet HEAD`).
+   The TS fix + os-audit.sh are committed & pushed; version-bump artifacts reverted to 1.7.90.
+2. Re-run the release: `scripts/create-release.sh 1.7.91-alpha`. Backend is cached (only a .ts
+   changed) so it's fast; the frontend build now passes. It bumps versions, builds, writes
+   releases/manifest.json (→1.7.91-alpha), commits, and tags v1.7.91-alpha.
+   - Memory guards: grep the staged frontend tarball for "1.7.91-alpha" before shipping (silent
+     vue-tsc failures); tarball must be flat (`tar -C web/dist/neode-ui .`).
+3. Publish: `scripts/publish-release-assets.sh 1.7.91-alpha gitea-vps2`, then
+   `git push origin main && git push origin --tags` (origin pushes to BOTH gitea-local + vps2).
+4. Verify manifest LIVE (this is "published"):
+   `curl -fsS http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json | jq .version`
+   must show `1.7.91-alpha`. **Then notify the user — they asked to be told when 1.7.91 publishes.**
+5. os-audit harness: run a full green pass on .116
+   (`ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http ARCHY_PASSWORD='ThisIsWeb54321@' tests/lifecycle/os-audit.sh`),
+   confirm Section A FM12 now reads `update_in_progress=false` (PASS not WARN), review B + C findings,
+   then wire os-audit.sh into the reboot-survival (L3) loop as the per-boot gate.
+
+---
+
+# ─ HISTORY — v1.7.89-alpha pass (2026-06-12), superseded ─
+
+Last updated: 2026-06-12 ~17:45 EDT (session on node .116)
+
+## RESUME PROMPT (paste into a fresh session)
+
+> Continue the v1.7.89-alpha release pass from /home/archipelago/Projects/archy on node .116.
+> Read docs/WEEKLY_RELEASE_TRACKER.md fully first — it has root causes, fixes already made,
+> and exact next steps. Do NOT redo: AIUI revert (done, validated), updater fixes in
+> core/archipelago/src/update.rs (done, uncommitted), .116 OTA unwedge (done). Resume at
+> "EXACT NEXT STEPS" below.
+
+## EXACT NEXT STEPS (in order)
+
+1. Backend focused tests were running in background:
+   `cd core && timeout 1500 cargo test -p archipelago -- update:: lnd container::image_versions scanner`
+   (log: /tmp/claude-.../tasks/bds4jk19e.output — if lost, just rerun the command; first
+   attempt died at 400s timeout during test compile, 1500s is the right budget).
+   Need: all green.
+2. RESOLVED before session end: vitest recheck passed clean — EXIT=0, 79 files / 645 tests,
+   even while cargo test was compiling. The earlier harness ui-unit-tests FAIL was load/flake
+   (machine saturated by the parallel cargo test compile), not a real failure. On resume just
+   rerun `tests/release/run.sh --quick` WITHOUT a parallel cargo build to confirm green;
+   if it ever fails again, the failing test name is in the stage output (drop `--silent`).
+3. Run full harness: `tests/release/run.sh` (static+frontend+backend). Then commit ALL
+   working-tree changes (one commit, e.g. "fix: harden OTA updates, AIUI desktop gap, LND
+   no-proxy" — CHANGELOG v1.7.89 section is already curated).
+4. Cut release: `scripts/create-release.sh 1.7.89-alpha` (needs clean tree, on main,
+   validates CHANGELOG section exists — it does). Then
+   `tests/release/run.sh --manifest` should pass, and grep the staged frontend tarball
+   for 1.7.89-alpha (memory: silent build failures).
+5. Publish: `scripts/publish-release-assets.sh 1.7.89-alpha gitea-vps2`, then
+   `git push origin main && git push origin --tags` and push gitea-local + tags too.
+   Verify manifest live on http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json
+6. Verify OTA on THIS node (.116): schedule is auto_apply; either wait for the scheduler
+   or trigger via UI. Confirm /var/lib/archipelago/update_state.json current_version
+   becomes 1.7.89-alpha, `update_in_progress` returns to false, web-ui + binary versions
+   MATCH (this node currently has web-ui 1.7.84 / binary 1.7.85 mismatch — the OTA heals it),
+   and journalctl shows "Post-OTA verification succeeded" (the new probe falls back to
+   http://127.0.0.1/ which is what .116 serves).
+7. Update this tracker + docs/PROGRESS_MEMORY.md, mark tasks done.
+Purpose: live tracker for this pass — test everything shipped this week (v1.7.83→v1.7.89),
+build the release test harness, fix OTA updates on .116, make updates bulletproof, cut v1.7.89-alpha.
+If the session is cut off, resume from here.
+
+## Task status
+
+| # | Task | Status |
+|---|------|--------|
+| 1 | AIUI revert (mobile back/close gone, desktop gap fixed) | DONE — validated |
+| 2 | Dev server on :8100 with embedded AIUI | DONE — see below |
+| 3 | Inventory this week's release-log items | DONE — see checklist |
+| 4 | Test harness covering this week + seed of system-wide harness | IN PROGRESS |
+| 5 | Fix OTA updates on .116 + bulletproof updates | IN PROGRESS — diagnosis below |
+| 6 | Cut v1.7.89-alpha release | PENDING (gates: 4, 5) |
+
+## State of the working tree
+
+- HEAD = 495b9078 (v1.7.89 changelog + AIUI mobile restore committed).
+- Uncommitted, intended for v1.7.89-alpha:
+  - `neode-ui/src/views/Dashboard.vue` — chat route back to plain `h-full` (desktop bottom-gap fix). Validated.
+  - `core/.../rpc/lnd/*` + `container/lnd.rs` — LND REST no-proxy + wallet readiness/unlock fixes.
+  - Version bumps to 1.7.89-alpha (Cargo.toml, package.json, locks), CHANGELOG entry.
+  - `neode-ui/vite.config.ts` — added `/aiui` dev proxy (keep; dev-only convenience).
+
+## AIUI validation (task 1) — DONE
+
+- HEAD already removed the mobile back button and restored `hideClose=true` (495b9078).
+- Working-tree Dashboard.vue removes `dashboard-scroll-panel mobile-scroll-pad` from the chat
+  route (that padding caused the desktop bottom gap); mesh keeps its styling.
+- Chat CSS verified byte-identical to last-good 34c4e87d (May 20).
+- Playwright check (desktop 1440x900, mobile 390x844): chat fills full viewport, no bottom gap,
+  no mobile back/close. `npm run type-check` + focused route tests + full vitest (645/645) pass.
+
+## Dev server on :8100 (task 2) — DONE
+
+- Running: `BACKEND_URL=http://127.0.0.1:5678 VITE_AIUI_URL=/aiui/ npx vite --host 0.0.0.0 --port 8100`
+  from `neode-ui/` (real local backend on 5678).
+- AIUI now embeds in /dashboard/chat via new vite proxy `/aiui` → `http://127.0.0.1:80`
+  (the node's deployed AIUI), same-origin like production.
+- Secondary throwaway instance for automated checks: :8101 against mock backend
+  (`node mock-backend.js` on 5959, password `password123`).
+
+## This week's shipped items (v1.7.83 → v1.7.89) — test checklist
+
+### Frontend (vitest/type-check/build cover most; full suite 645/645 green 2026-06-12)
+- [x] AIUI fast launch, no availability probe (v1.7.88) — covered by visual check + Chat.vue tests
+- [x] AIUI mobile layout restore (v1.7.89) — playwright visual check
+- [x] App-session launch metadata from manifests / typed interfaces (v1.7.83) — appSessionConfig tests
+- [x] OnlyOffice + Saleor removal (v1.7.83) — catalog tests
+- [ ] Bitcoin receive UI flow end-to-end (v1.7.87/88) — needs live LND node check
+- [ ] Fleet tab keeps node list/alerts during refresh, names not hashes (v1.7.85/86) — store tests?
+- [ ] Credential interstitial full-screen overlay (v1.7.87) — visual
+- [ ] Mobile federation/system-update buttons full width (v1.7.86) — visual
+
+### Backend (cargo)
+- [ ] LND REST no-proxy client + GET newaddress p2wkh (v1.7.88/89) — unit tests + live check
+- [ ] LND wallet readiness/unlock after restart (v1.7.89) — unit + live
+- [ ] Bitcoin trusted-node relay rpcauth/txrelay (v1.7.84) — unit tests exist? check
+- [ ] Container scanner RAII in-flight guard (v1.7.84) — cargo test
+- [ ] ElectrumX health-check startup window + cache tuning (v1.7.85/86)
+- [ ] Portainer pin 2.19.4 / bitcoin-ui image pin (v1.7.84/85) — image-versions tests
+- [ ] Fleet telemetry name/hostname/URL fields (v1.7.85)
+- [ ] Federation no self-import (v1.7.85)
+- [ ] Kiosk safe-area + self-update refreshes kiosk files (v1.7.84)
+- [ ] Wi-Fi scan error/retry/escaped SSID/open networks (v1.7.84)
+
+### OTA / updates (task 5)
+- [ ] .116 stuck: current 1.7.85-alpha, `update_in_progress: true` since 1.7.88 attempt — diagnose+fix
+- [ ] Updater hardening: stuck-in-progress recovery, resumable/atomic apply, verify post-restart version
+
+## OTA diagnosis on .116 — ROOT CAUSES FOUND + FIXED (code staged for v1.7.89)
+
+Four bugs, all reproduced from the journal (Jun 12 03:45–04:33):
+
+1. Post-OTA probe only tries `https://127.0.0.1/`; .116's nginx binds only :80 (443 is
+   tailscale's) → connection refused × 18 → a GOOD 1.7.85 update was "rolled back".
+   FIX: probe falls back to `http://127.0.0.1/` on connect error (update.rs probe_frontend_once).
+2. That rollback's binary restore did `host_sudo cp` onto the RUNNING binary → ETXTBSY exit 1
+   → binary stayed 1.7.85 while web-ui rolled back to 1.7.84 (mismatch confirmed live).
+   FIX: rollback now cp→tmp→atomic mv, same pattern as apply (update.rs rollback_update).
+3. The rollback chown'd `update-backup/archipelago` root:root IN PLACE → next apply's
+   fs::copy (as service user) hit EACCES → "Failed to backup current binary" × 3 → 1.7.86/88
+   never applied. FIX: apply unlinks stale backup first; rollback chowns only its temp copy.
+4. Failed apply left `update_in_progress: true` wedged (staging still populated so the
+   stale-flag guard never fires). Unwedged operationally; fixed structurally by 1–3.
+
+Operational cleanup DONE on .116 (2026-06-12 17:15): removed root-owned
+`update-backup/archipelago`, stale `update-staging/` (1.7.86), and the stale
+`update-pending-verify.json`. Next state load clears `update_in_progress`.
+NOTE: live web-ui is 1.7.84 / binary 1.7.85 (mismatch from bug 2). Not hand-patched —
+the v1.7.89 OTA will resync both. Good 1.7.85 frontend is quarantined at
+`/opt/archipelago/web-ui.failed.1781250438247`.
+Verification plan: after v1.7.89 release, watch .116 auto-apply (schedule auto_apply),
+confirm `update_state.json.current_version == 1.7.89-alpha` and web-ui version matches.
+
+## Test harness (task 4) — CREATED at tests/release/run.sh
+
+- Stages: static (git diff --check, cargo fmt, catalog drift, optional --manifest),
+  frontend (type-check, full vitest), optional --with-build (build + grep dist for version),
+  backend (cargo check + focused cargo test: update:: lnd container::image_versions scanner,
+  all wrapped in `timeout`), optional --live URL smoke (/, /aiui/, /rpc/v1).
+- Results so far (2026-06-12): type-check PASS, full vitest 645/645 PASS, cargo fmt PASS,
+  cargo check PASS, catalog drift PASS (3 pre-existing MISSING_CATALOG warnings, exit 0,
+  identical on HEAD). Focused backend cargo tests running (first run hit the known slow
+  test-compile on .116 at 400s timeout; rerunning with 1500s).
+- AIUI embed verified end-to-end via playwright on :8101 (mock backend): iframe loads,
+  `ready` handshake clears the loading overlay, hideClose honored.
+- Release flow confirmed: commit all → `scripts/create-release.sh 1.7.89-alpha` (validates
+  curated CHANGELOG section, builds, manifests, commits, tags) →
+  `scripts/publish-release-assets.sh 1.7.89-alpha gitea-vps2` → push origin main + tags.
+  Tarball layout/perms safety is already inside create-release-manifest.sh.
+- CHANGELOG v1.7.89 section rewritten layman-readable (updater fixes added).
+
+## Release gates for v1.7.89-alpha (task 6)
+
+1. All harness stages green locally.
+2. OTA fix for stuck `update_in_progress` included + .116 updates successfully to the new release.
+3. Frontend build: grep packaged tarball for "1.7.89-alpha" before shipping (memory: silent vue-tsc failures).
+4. Flat tarball layout (`tar -C web/dist/neode-ui .`).
+5. Commit, tag `v1.7.89-alpha`, push origin + gitea-local + tags, publish release assets, verify
+   manifest + node OTA picks it up.

neode-ui/src/utils/__tests__/bitcoinReceive.test.ts

@@ -21,4 +21,40 @@ describe('explainReceiveAddressFailure', () => {
   it('keeps bitcoin address generation failures visible', () => {
     expect(explainReceiveAddressFailure(new Error('Bitcoin address generation failed: LND is not ready'))).toContain('Bitcoin address generation failed')
   })
+
+  describe('structured reason codes (backend [CODE] token)', () => {
+    it('maps an unreachable REST endpoint to a starting/recovering message — NOT locked (.228 regression)', () => {
+      const msg = explainReceiveAddressFailure(
+        new Error('Bitcoin address unavailable [LND_REST_UNREACHABLE]: service not reachable'),
+      )
+      expect(msg).toContain('starting up or recovering')
+      expect(msg.toLowerCase()).not.toContain('locked')
+    })
+
+    it('maps a genuinely locked wallet to the locked message', () => {
+      expect(
+        explainReceiveAddressFailure(new Error('Bitcoin address unavailable [LND_WALLET_LOCKED]: locked')),
+      ).toContain('locked')
+    })
+
+    it('maps an uninitialized wallet to the setup message', () => {
+      expect(
+        explainReceiveAddressFailure(new Error('Bitcoin address unavailable [LND_WALLET_UNINITIALIZED]: x')),
+      ).toContain('has not been set up')
+    })
+
+    it('maps a syncing wallet to the syncing message', () => {
+      expect(
+        explainReceiveAddressFailure(new Error('Bitcoin address unavailable [LND_SYNCING]: x')),
+      ).toContain('syncing')
+    })
+
+    it('prefers the code over substrings even when the detail text is misleading', () => {
+      // Detail mentions neither "locked" nor "unlock"; code must still win.
+      const msg = explainReceiveAddressFailure(
+        new Error('Bitcoin address unavailable [LND_REST_UNREACHABLE]: wallet service down'),
+      )
+      expect(msg).toContain('starting up or recovering')
+    })
+  })
 })

neode-ui/src/utils/bitcoinReceive.ts

@@ -1,5 +1,29 @@
+// Machine-readable reason codes the backend embeds as a `[CODE]` token in
+// receive-address errors (see core .../api/rpc/lnd/wallet.rs). Mapping the code
+// directly is precise — unlike the substring heuristics below, it cannot
+// mislabel an unreachable-REST failure as "wallet is locked" (the .228 bug).
+const RECEIVE_CODE_MESSAGES: Record<string, string> = {
+  LND_REST_UNREACHABLE:
+    'Bitcoin address is not ready yet because the Lightning wallet service is still starting up or recovering. Please try again in a moment.',
+  LND_WALLET_LOCKED:
+    'Bitcoin address is not ready because the Lightning wallet is locked. Unlock or initialize LND first.',
+  LND_WALLET_UNINITIALIZED:
+    'Bitcoin address is not ready because the Lightning wallet has not been set up yet. Finish wallet setup, then try again.',
+  LND_SYNCING:
+    'Bitcoin address is not ready while the wallet is still syncing with the Bitcoin network. Try again once sync has progressed.',
+  LND_ERROR:
+    'Bitcoin address is not ready yet. Check that the Lightning app is healthy, then try again.',
+}
+
 export function explainReceiveAddressFailure(error: unknown): string {
   const message = error instanceof Error ? error.message : String(error || '')
+
+  // Prefer the structured reason code when present.
+  const code = message.match(/\[([A-Z_]+)\]/)?.[1]
+  if (code && RECEIVE_CODE_MESSAGES[code]) {
+    return RECEIVE_CODE_MESSAGES[code]
+  }
+
   const lower = message.toLowerCase()
 
   if (lower.includes('wallet') && (lower.includes('locked') || lower.includes('unlock'))) {

neode-ui/src/views/appSession/generatedAppSessionConfig.ts

@@ -7,6 +7,7 @@ export const GENERATED_APP_PORTS: Record<string, number> = {
   "botfights": 9100,
   "btcpay-server": 23000,
   "did-wallet": 8083,
+  "electrumx": 50002,
   "fedimint": 8175,
   "filebrowser": 8083,
   "gitea": 3001,

neode-ui/src/views/apps/appsConfig.ts

@@ -189,7 +189,14 @@ export function canLaunch(pkg: PackageDataEntry): boolean {
   if ((pkg.manifest.id === 'fedimint' || pkg.manifest.id === 'fedimintd') && hasUI) {
     return pkg.state === PackageState.Running || pkg.state === PackageState.Starting
   }
-  return !!hasUI && pkg.state === 'running' && pkg.health !== 'starting' && pkg.health !== 'unhealthy'
+  // A static launch URL (e.g. a host-networked companion UI like
+  // archy-electrs-ui) serves independently of the backend's own sync state, so
+  // the tile stays launchable while the backend is still 'starting' (ElectrumX
+  // indexes for 10m+ on first run). A genuinely 'unhealthy' backend still
+  // blocks. Apps that rely on a runtime interface-address keep the strict gate.
+  const blockedByHealth =
+    pkg.health === 'unhealthy' || (pkg.health === 'starting' && !hasKnownLaunchUrl)
+  return !!hasUI && pkg.state === 'running' && !blockedByHealth
 }
 
 export function launchBlockedReason(id: string, pkg?: PackageDataEntry | null): string {

scripts/create-release.sh

@@ -75,6 +75,28 @@ if ! git -C "$PROJECT_ROOT" diff --quiet HEAD; then
     exit 1
 fi
 
+# ── Pre-flight test gate ──────────────────────────────────────────────
+# A release must not ship if the static/frontend/backend checks fail. This
+# runs the release gate harness (cargo fmt/check, catalog drift, vitest, and
+# the focused cargo suites — incl. the receive/port-drift/secret regressions).
+# Skipped on --dry-run, or set SKIP_RELEASE_TESTS=1 to bypass in an emergency.
+# The lifecycle bats harness (tests/lifecycle/run-20x.sh) still runs separately
+# against live nodes — see tests/lifecycle/TESTING.md.
+if ! $DRY_RUN; then
+    if [ "${SKIP_RELEASE_TESTS:-0}" = "1" ]; then
+        echo "WARNING: SKIP_RELEASE_TESTS=1 — bypassing the pre-flight test gate"
+    elif [ -x "$PROJECT_ROOT/tests/release/run.sh" ]; then
+        echo "[0/7] Running release gate (tests/release/run.sh)..."
+        if ! "$PROJECT_ROOT/tests/release/run.sh"; then
+            echo "Error: release gate failed — aborting release. Fix the failing"
+            echo "       stage, or re-run with SKIP_RELEASE_TESTS=1 to override."
+            exit 1
+        fi
+    else
+        echo "WARNING: tests/release/run.sh not found/executable — skipping test gate"
+    fi
+fi
+
 # Check tag doesn't already exist
 if git -C "$PROJECT_ROOT" tag -l "v$VERSION" | grep -q "v$VERSION"; then
     echo "Error: Tag v$VERSION already exists"
@@ -94,6 +116,7 @@ echo ""
 
 if $DRY_RUN; then
     echo "[DRY RUN] Would perform the following:"
+    echo "  0. Run pre-flight test gate (tests/release/run.sh) — aborts on failure"
     echo "  1. Update core/archipelago/Cargo.toml version to $VERSION"
     echo "  2. Update neode-ui/package.json version to $VERSION"
     echo "  3. Build backend (cargo build --release -p archipelago)"

tests/lifecycle/TESTING.md

@@ -58,10 +58,29 @@ v1.7.52 tags.
 | L1 RPC | 70 | bitcoin-knots, lnd, electrumx, btcpay, mempool, fedimint, required-stack, package-update-smoke | ● for the 6 core apps |
 | L2 UI | 9 | ui-coverage | ● for dashboard + 7 proxy paths + bitcoin-ui:8334 |
 | L3 lifecycle survival | 14 | companion-survives-archipelago-restart, backend-survives-archipelago-restart, required-stack-destructive, use-quadlet-backends-install | ◐ companions ● ; backends ◐ regression-gate (will fail until Phase 3 Quadlet ships); quadlet post-condition gate ✅ skip-clean today, hard gate when flag flipped |
+| L1 wallet-receive / drift / secrets | 5 | bitcoin-receive, port-drift, secret-completeness | ● guards the v1.7.9x wallet fleet failures |
 | L4 browser journey | 0 | none | ○ not started |
 | L5 chaos | 0 | none | ○ not started |
 | L6 performance | 0 | none | ○ not started |
 
+### Wallet / Bitcoin fleet-failure regression suites (added after v1.7.90-alpha)
+
+Three production failures shipped on v1.7.90-alpha despite the existing harness,
+because nothing exercised the receive path, port-mapping drift, or secret
+completeness on a live node. New suites close those gaps (all run on the archy
+host, read-only, so they join `run.sh`/`run-20x.sh` automatically):
+
+| Suite | Failure it guards | Asserts |
+|---|---|---|
+| `bitcoin-receive.bats` | .116 ("Operation failed" on receive) and .228 (false "wallet is locked") | LND REST reachable on the **manifest** host port; `lnd.newaddress` returns a `bc1…` address on a running node; receive errors are specific, never the generic catch-all |
+| `port-drift.bats` | .116 (lnd REST stuck on host 8080 vs manifest 18080) | every installed backend's live `podman inspect` PortBindings match its manifest `ports:` (the external mirror of the orchestrator's `host_port_bindings_drifted`) |
+| `secret-completeness.bats` | .198 (bitcoin-knots needs `bitcoin-rpc-txrelay-rpcauth`, never generated → stack cascade) | every `secret_file` referenced by an installed backend manifest exists in the secrets dir |
+
+Backed by L0 unit tests (`cargo test … drift missing_secret lnd`) and a vitest
+for the frontend reason-code mapping (`bitcoinReceive.test.ts`). The release
+gate `scripts/create-release.sh` now runs `tests/release/run.sh` (which includes
+these) and **aborts the release on failure** — previously it ran no tests at all.
+
 ## Run commands
 
 ```bash

tests/lifecycle/bats/bitcoin-receive.bats

@@ -0,0 +1,104 @@
+#!/usr/bin/env bats
+# tests/lifecycle/bats/bitcoin-receive.bats
+#
+# Regression coverage for the Bitcoin "Receive" flow. Receive addresses come
+# from LND's hot wallet via the `lnd.newaddress` RPC, so this exercises the
+# exact path that broke on the fleet:
+#   - .116: LND REST published on the wrong host port (8080 vs the manifest's
+#           18080) -> connection refused -> receive failed with the generic
+#           "Operation failed. Check server logs." message.
+#   - .228: the same family surfaced to the UI as a *false* "wallet is locked".
+#
+# These tests run on the archy host (they shell into podman / curl localhost).
+#
+# Tiers: read-only only — generating a receive address is non-destructive.
+
+load '../lib/rpc.bash'
+
+setup_file() {
+  : "${ARCHY_PASSWORD:?Set ARCHY_PASSWORD env var to the UI password}"
+  export ARCHY_FORCE_LOGIN=1
+  rpc_login
+  unset ARCHY_FORCE_LOGIN
+}
+
+teardown_file() {
+  rpc_logout_local
+}
+
+# Resolve the LND REST host port from the manifest (single source of truth) so
+# this test follows the manifest rather than hard-coding 18080.
+_lnd_rest_host_port() {
+  local mf
+  for mf in \
+    "${ARCHIPELAGO_APPS_DIR:-/opt/archipelago/apps}/lnd/manifest.yml" \
+    "${ARCHIPELAGO_APPS_DIR:-/opt/archipelago/apps}/lnd/manifest.yaml" \
+    "$BATS_TEST_DIRNAME/../../../apps/lnd/manifest.yml"; do
+    [[ -r "$mf" ]] || continue
+    # The REST mapping is the `- host: <N>` whose following `container:` is 8080.
+    awk '
+      /- host:/        { host=$3 }
+      /container:/     { if ($2 == 8080 && host != "") { print host; exit } }
+    ' "$mf"
+    return 0
+  done
+}
+
+_lnd_running() {
+  rpc_result container-list 2>/dev/null \
+    | jq -e '.[] | select(.name == "lnd" and .state == "running")' >/dev/null 2>&1
+}
+
+# ────────────────────────────────────────────────────────────────────
+# Read-only tier
+# ────────────────────────────────────────────────────────────────────
+
+@test "LND REST is reachable on the manifest host port (catches port drift)" {
+  _lnd_running || skip "lnd not running"
+  local port
+  port=$(_lnd_rest_host_port)
+  [[ -n "$port" ]] || skip "could not resolve LND REST host port from manifest"
+
+  # A TCP connect is enough: drift (container published on a different host
+  # port) shows up as connection-refused here, exactly as on .116.
+  run curl -sk -o /dev/null --max-time 8 "https://127.0.0.1:${port}/v1/getinfo"
+  if [ "$status" -ne 0 ]; then
+    echo "LND REST not reachable on host port ${port} (curl exit $status) — likely published-port drift" >&2
+    return 1
+  fi
+}
+
+@test "lnd.newaddress returns a bech32 address when lnd is running" {
+  _lnd_running || skip "lnd not running"
+
+  run rpc_call lnd.newaddress
+  [ "$status" -eq 0 ]
+
+  local err addr
+  err=$(echo "$output" | jq -r '.error.message // .error // empty')
+  addr=$(echo "$output" | jq -r '.result.address // empty')
+
+  # The whole point of the fix: a running lnd must hand back a real address.
+  if [[ -n "$err" ]]; then
+    echo "lnd.newaddress errored on a running node: $err" >&2
+    return 1
+  fi
+  if [[ "$addr" != bc1* ]]; then
+    echo "expected a bech32 (bc1…) address, got: '$addr'" >&2
+    return 1
+  fi
+}
+
+@test "receive errors are specific, never the generic catch-all" {
+  # Even when receive legitimately can't produce an address, the message must be
+  # actionable (start with 'Bitcoin address' and/or carry a [CODE] token) — the
+  # generic 'Operation failed' is what hid the real cause on .116.
+  run rpc_call lnd.newaddress
+  [ "$status" -eq 0 ]
+  local err
+  err=$(echo "$output" | jq -r '.error.message // .error // empty')
+  if [[ "$err" == "Operation failed. Check server logs for details." ]]; then
+    echo "receive returned the generic catch-all instead of a specific reason" >&2
+    return 1
+  fi
+}

tests/lifecycle/bats/port-drift.bats

@@ -0,0 +1,78 @@
+#!/usr/bin/env bats
+# tests/lifecycle/bats/port-drift.bats
+#
+# Regression guard for the .116 failure class: a backend container that is
+# "Up" but publishes its ports to the WRONG host ports because the manifest
+# changed after the container was created (e.g. lnd REST stuck on host 8080
+# while the manifest — and every in-process client — expects 18080).
+#
+# This mirrors the orchestrator's `host_port_bindings_drifted` check, but from
+# the outside: it compares the live `podman inspect` PortBindings against the
+# manifest `ports:` for each installed backend. Runs on the archy host.
+#
+# Tiers: read-only.
+
+_apps_dir() {
+  local d
+  for d in "${ARCHIPELAGO_APPS_DIR:-}" /opt/archipelago/apps \
+           "$BATS_TEST_DIRNAME/../../../apps"; do
+    [[ -n "$d" && -d "$d" ]] && { echo "$d"; return 0; }
+  done
+  return 1
+}
+
+_manifest_for() {
+  local app="$1" dir
+  dir=$(_apps_dir) || return 1
+  local mf
+  for mf in "$dir/$app/manifest.yml" "$dir/$app/manifest.yaml"; do
+    [[ -r "$mf" ]] && { echo "$mf"; return 0; }
+  done
+  return 1
+}
+
+# Emit "host container" pairs from a manifest's ports: block.
+_manifest_ports() {
+  awk '
+    /^[[:space:]]*ports:/ { inports=1; next }
+    inports && /^[[:space:]]*[a-z_]+:[[:space:]]*$/ && !/protocol:|host:|container:/ { inports=0 }
+    inports && /- host:/    { host=$3 }
+    inports && /container:/  { print host, $2 }
+  ' "$1"
+}
+
+# For a given container + (host,container) port, emit a "DRIFT: …" line on
+# mismatch (and nothing otherwise). Stays silent for unpublished / host-net
+# ports — those are handled elsewhere and must never be treated as drift.
+_drift_line() {
+  local cname="$1" want_host="$2" cport="$3"
+  local bindings actual
+  bindings=$(podman inspect "$cname" --format '{{json .HostConfig.PortBindings}}' 2>/dev/null) || return 0
+  actual=$(echo "$bindings" | jq -r --arg k "${cport}/tcp" '.[$k][]?.HostPort // empty' 2>/dev/null)
+  [[ -n "$actual" ]] || return 0
+  echo "$actual" | grep -qx "$want_host" && return 0
+  echo "DRIFT: $cname container-port $cport published on host [$actual] but manifest wants $want_host"
+}
+
+@test "backend containers publish ports that match their manifest" {
+  command -v podman >/dev/null 2>&1 || skip "podman not available"
+  local checked=0 violations="" app cname mf line
+  # container-name : manifest-app-id
+  for pair in "lnd:lnd" "bitcoin-knots:bitcoin-knots" "electrumx:electrumx"; do
+    cname="${pair%%:*}"; app="${pair##*:}"
+    podman container exists "$cname" 2>/dev/null || continue
+    mf=$(_manifest_for "$app") || continue
+    while read -r host cport; do
+      [[ -n "$host" && -n "$cport" ]] || continue
+      checked=$((checked + 1))
+      line=$(_drift_line "$cname" "$host" "$cport")
+      [[ -n "$line" ]] && violations+="${line}"$'\n'
+    done < <(_manifest_ports "$mf")
+  done
+  [[ "$checked" -gt 0 ]] || skip "no installed backend containers with published ports to check"
+  if [[ -n "$violations" ]]; then
+    echo "published-port drift detected:" >&2
+    echo "$violations" >&2
+    return 1
+  fi
+}

tests/lifecycle/bats/secret-completeness.bats

@@ -0,0 +1,73 @@
+#!/usr/bin/env bats
+# tests/lifecycle/bats/secret-completeness.bats
+#
+# Regression guard for the .198 failure class: a manifest references a
+# `secret_env.secret_file` that was never generated on the node, so secret
+# resolution hard-fails and the container won't start — cascading the whole
+# Bitcoin stack. (bitcoin-knots gained `bitcoin-rpc-txrelay-rpcauth`, which old
+# nodes lacked, so bitcoind never came up and reinstall "just stopped".)
+#
+# For every installed backend, assert every secret_file it references exists in
+# the secrets dir. Runs on the archy host.
+#
+# Tiers: read-only.
+
+SECRETS_DIR="${ARCHY_SECRETS_DIR:-/var/lib/archipelago/secrets}"
+
+_apps_dir() {
+  local d
+  for d in "${ARCHIPELAGO_APPS_DIR:-}" /opt/archipelago/apps \
+           "$BATS_TEST_DIRNAME/../../../apps"; do
+    [[ -n "$d" && -d "$d" ]] && { echo "$d"; return 0; }
+  done
+  return 1
+}
+
+_manifest_for() {
+  local app="$1" dir mf
+  dir=$(_apps_dir) || return 1
+  for mf in "$dir/$app/manifest.yml" "$dir/$app/manifest.yaml"; do
+    [[ -r "$mf" ]] && { echo "$mf"; return 0; }
+  done
+  return 1
+}
+
+_secret_files_in() {
+  # Emit each `secret_file:` value referenced by the manifest.
+  grep -E '^[[:space:]]*secret_file:' "$1" 2>/dev/null | awk '{print $2}'
+}
+
+_secret_exists() {
+  local f="$SECRETS_DIR/$1"
+  [[ -e "$f" ]] && return 0
+  sudo -n test -f "$f" 2>/dev/null
+}
+
+@test "every installed backend's referenced secrets exist on disk" {
+  command -v podman >/dev/null 2>&1 || skip "podman not available"
+  [[ -d "$SECRETS_DIR" ]] || sudo -n test -d "$SECRETS_DIR" 2>/dev/null || skip "secrets dir not present"
+
+  local checked=0 missing="" app cname mf sf
+  # container-name : manifest-app-id  (the bitcoin stack that cascades)
+  for pair in \
+    "bitcoin-knots:bitcoin-knots" "lnd:lnd" "electrumx:electrumx" \
+    "mempool-api:mempool-api" "btcpay-server:btcpay-server" \
+    "archy-nbxplorer:archy-nbxplorer" "fedimint:fedimint" \
+    "fedimint-gateway:fedimint-gateway"; do
+    cname="${pair%%:*}"; app="${pair##*:}"
+    podman container exists "$cname" 2>/dev/null || continue
+    mf=$(_manifest_for "$app") || continue
+    while read -r sf; do
+      [[ -n "$sf" ]] || continue
+      checked=$((checked + 1))
+      _secret_exists "$sf" || missing+="${app} -> ${sf}\n"
+    done < <(_secret_files_in "$mf")
+  done
+
+  [[ "$checked" -gt 0 ]] || skip "no installed backends with secret references to check"
+  if [[ -n "$missing" ]]; then
+    echo "installed apps reference missing secrets:" >&2
+    echo -e "$missing" >&2
+    return 1
+  fi
+}

tests/lifecycle/os-audit.sh

@@ -0,0 +1,239 @@
+#!/usr/bin/env bash
+# tests/lifecycle/os-audit.sh — one non-destructive OS-wide health gate.
+#
+# Ties together, in a single pass with one scorecard + exit code:
+#   A. Backend / RPC health      — node is up, not wedged mid-OTA, core daemons answer
+#   B. All-apps lifecycle audit  — every catalog app: valid state, real health,
+#                                  reachable launch URL, populated launch metadata
+#                                  (delegates to remote-lifecycle.sh, audit-only)
+#   C. FM-guards                 — the concrete failure modes that have bitten the
+#                                  fleet: port-drift (FM8), secret-completeness (FM2),
+#                                  orphaned container states (FM9), OTA wedge (FM12)
+#
+# Everything here is READ-ONLY: no install/stop/start/uninstall, no service bounce.
+# Safe to run against a live production node. It is the per-boot building block the
+# reboot-survival harness (L3) calls after each reboot.
+#
+# Env:
+#   ARCHY_HOST       (default 127.0.0.1)
+#   ARCHY_SCHEME     (default https; use http for .116 / nginx-:80-only nodes)
+#   ARCHY_PASSWORD   (required)
+#   ARCHY_LOCAL      (auto: 1 when ARCHY_HOST is loopback) — gates host-only podman checks
+#
+# Usage:
+#   ARCHY_HOST=127.0.0.1 ARCHY_SCHEME=http ARCHY_PASSWORD=... tests/lifecycle/os-audit.sh
+#
+# Exit: 0 = every section green; 1 = one or more checks failed; 2 = setup/usage error.
+
+set -uo pipefail
+
+HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+
+ARCHY_HOST="${ARCHY_HOST:-127.0.0.1}"
+ARCHY_SCHEME="${ARCHY_SCHEME:-https}"
+ARCHY_PASSWORD="${ARCHY_PASSWORD:-}"
+BASE_URL="${ARCHY_SCHEME}://${ARCHY_HOST}"
+
+# Host-only checks (podman sweeps) make sense only when this script runs ON the node.
+if [[ -z "${ARCHY_LOCAL:-}" ]]; then
+  case "$ARCHY_HOST" in
+    127.0.0.1|localhost|::1) ARCHY_LOCAL=1 ;;
+    *) ARCHY_LOCAL=0 ;;
+  esac
+fi
+
+if [[ -z "$ARCHY_PASSWORD" ]]; then
+  echo "ARCHY_PASSWORD env var must be set." >&2
+  exit 2
+fi
+for tool in curl jq; do
+  command -v "$tool" >/dev/null 2>&1 || { echo "missing required tool: $tool" >&2; exit 2; }
+done
+
+# ── scorecard state ───────────────────────────────────────────────────────────
+PASS=0; FAIL=0; WARN=0
+declare -a RESULTS=()
+record() { # record <PASS|FAIL|WARN> <label> [detail]
+  local status="$1" label="$2" detail="${3:-}"
+  case "$status" in
+    PASS) PASS=$((PASS+1)) ;;
+    FAIL) FAIL=$((FAIL+1)) ;;
+    WARN) WARN=$((WARN+1)) ;;
+  esac
+  RESULTS+=("$(printf '%-4s %-38s %s' "$status" "$label" "$detail")")
+  printf '  [%s] %s %s\n' "$status" "$label" "$detail"
+}
+
+# ── minimal RPC client (session + CSRF) ────────────────────────────────────────
+SESSION=""; CSRF=""
+rpc_login() {
+  local hdr; hdr=$(mktemp)
+  curl -sk -D "$hdr" -X POST "${BASE_URL}/rpc/v1" -H 'Content-Type: application/json' \
+    -d "$(jq -nc --arg p "$ARCHY_PASSWORD" '{jsonrpc:"2.0",id:1,method:"auth.login",params:{password:$p}}')" \
+    -o /dev/null 2>/dev/null
+  SESSION=$(grep -i '^set-cookie: session=' "$hdr" | head -1 | sed -E 's/.*session=([^;]+).*/\1/' | tr -d '\r')
+  CSRF=$(grep -i '^set-cookie: csrf_token=' "$hdr" | head -1 | sed -E 's/.*csrf_token=([^;]+).*/\1/' | tr -d '\r')
+  rm -f "$hdr"
+  [[ -n "$SESSION" && -n "$CSRF" ]]
+}
+# rpc <method> [params-json] -> prints raw JSON response
+rpc() {
+  local method="$1" params="${2:-{\}}"
+  curl -sk -X POST "${BASE_URL}/rpc/v1" -H 'Content-Type: application/json' \
+    -H "Cookie: session=${SESSION}; csrf_token=${CSRF}" -H "X-CSRF-Token: ${CSRF}" \
+    -d "$(jq -nc --arg m "$method" --argjson p "$params" '{jsonrpc:"2.0",id:2,method:$m,params:$p}')" 2>/dev/null
+}
+# rpc_ok <method> [params] -> 0 if a result came back with no error
+rpc_ok() {
+  local resp; resp=$(rpc "$@")
+  [[ -n "$resp" ]] && [[ "$(jq -r '.error // empty' <<<"$resp" 2>/dev/null)" == "" ]] \
+    && [[ "$(jq -r 'has("result")' <<<"$resp" 2>/dev/null)" == "true" ]]
+}
+
+# ══ Section A — Backend / RPC health ═══════════════════════════════════════════
+section_a() {
+  echo
+  echo "== A. Backend / RPC health =="
+
+  # unauth health probe first (doesn't need a session)
+  local health; health=$(curl -sk -X POST "${BASE_URL}/rpc/v1" -H 'Content-Type: application/json' \
+    -d '{"jsonrpc":"2.0","id":1,"method":"health","params":{}}' 2>/dev/null)
+  if [[ "$(jq -r '.result.status // empty' <<<"$health" 2>/dev/null)" =~ ^(ok|degraded)$ ]]; then
+    record PASS "node responds (health)" "status=$(jq -r '.result.status' <<<"$health")"
+  else
+    record FAIL "node responds (health)" "no/invalid health response — node down?"
+    return
+  fi
+
+  if ! rpc_login; then
+    record FAIL "auth.login" "could not establish session (wrong password or rate-limited)"
+    return
+  fi
+  record PASS "auth.login" "session established"
+
+  # FM12 — OTA must not be wedged mid-apply.
+  # NB: must use has() not `//` — jq's `//` treats a legit `false` as empty and
+  # would fall through to "unknown" on a perfectly healthy node.
+  local us; us=$(rpc update.status)
+  local inprog; inprog=$(jq -r '
+    if   (.result|type=="object") and (.result|has("update_in_progress")) then .result.update_in_progress
+    elif (.result|type=="object") and (.result|has("in_progress"))        then .result.in_progress
+    else "unknown" end' <<<"$us" 2>/dev/null)
+  if [[ "$inprog" == "false" ]]; then
+    record PASS "OTA not wedged (update.status)" "update_in_progress=false"
+  elif [[ "$inprog" == "unknown" ]]; then
+    record WARN "OTA not wedged (update.status)" "could not read update_in_progress"
+  else
+    record FAIL "OTA not wedged (update.status)" "update_in_progress=$inprog (FM12 wedge)"
+  fi
+
+  # Core daemons answer (only assert for ones present on this node)
+  if rpc_ok bitcoin.getinfo || rpc_ok bitcoin.relay-status; then
+    record PASS "bitcoin RPC reachable" ""
+  else
+    record WARN "bitcoin RPC reachable" "bitcoin.getinfo/relay-status did not answer (not installed?)"
+  fi
+  if rpc_ok lnd.getinfo; then
+    record PASS "lnd RPC reachable" ""
+  else
+    record WARN "lnd RPC reachable" "lnd.getinfo did not answer (not installed / wallet locked?)"
+  fi
+  if rpc_ok system.stats || rpc_ok system.get-metrics; then
+    record PASS "system metrics reachable" ""
+  else
+    record WARN "system metrics reachable" "system.stats/get-metrics did not answer"
+  fi
+
+  # FM13 — disk pressure early-warning (best-effort; field names vary by version)
+  local ds; ds=$(rpc system.disk-status)
+  local usep; usep=$(jq -r '[.result.use_percent,.result.used_percent,.result.percent]|map(select(.!=null))|first // empty' <<<"$ds" 2>/dev/null)
+  if [[ -n "$usep" ]]; then
+    if (( ${usep%.*} >= 90 )); then
+      record FAIL "disk pressure (system.disk-status)" "${usep}% used (FM13 risk)"
+    else
+      record PASS "disk pressure (system.disk-status)" "${usep}% used"
+    fi
+  fi
+}
+
+# ══ Section B — All-apps lifecycle audit (delegates to remote-lifecycle.sh) ═════
+section_b() {
+  echo
+  echo "== B. All-apps lifecycle audit (non-destructive, all catalog apps) =="
+  local out rc
+  # No ARCHY_APPS + no ARCHY_FULL_LIFECYCLE => audit every catalog app (audit_app).
+  out=$(ARCHY_HOST="$ARCHY_HOST" ARCHY_SCHEME="$ARCHY_SCHEME" ARCHY_PASSWORD="$ARCHY_PASSWORD" \
+        ARCHY_APPS="" ARCHY_FULL_LIFECYCLE=0 \
+        "$HERE/remote-lifecycle.sh" 2>&1)
+  rc=$?
+  # Surface the per-app lines but drop the noisy optional-probe jq parse errors.
+  echo "$out" | grep -vE '^jq: (parse )?error' | sed 's/^/    /'
+  if (( rc == 0 )); then
+    record PASS "broad all-apps audit" "remote-lifecycle.sh exit 0"
+  else
+    local n; n=$(echo "$out" | grep -oE 'FAILED checks: [0-9]+' | grep -oE '[0-9]+' | tail -1)
+    record FAIL "broad all-apps audit" "remote-lifecycle.sh exit $rc (${n:-?} app checks failed)"
+  fi
+}
+
+# ══ Section C — FM-guards ══════════════════════════════════════════════════════
+run_bats_guard() { # run_bats_guard <suite> <label> <fm>
+  local suite="$1" label="$2" fm="$3" out rc
+  if ! command -v bats >/dev/null 2>&1; then
+    record WARN "$label" "bats not installed — $fm guard skipped"
+    return
+  fi
+  out=$(ARCHY_HOST="$ARCHY_HOST" ARCHY_SCHEME="$ARCHY_SCHEME" ARCHY_PASSWORD="$ARCHY_PASSWORD" \
+        "$HERE/run.sh" "$suite" 2>&1); rc=$?
+  if (( rc == 0 )); then
+    record PASS "$label" "$fm guard green"
+  else
+    record FAIL "$label" "$fm — $(echo "$out" | grep -E '^not ok' | head -1)"
+  fi
+}
+
+section_c() {
+  echo
+  echo "== C. FM-guards (the concrete fleet failure modes) =="
+  run_bats_guard port-drift         "port bindings match manifest" "FM8"
+  run_bats_guard secret-completeness "all referenced secrets exist" "FM2"
+
+  # FM9 — orphaned container states (host-only: needs local podman)
+  if [[ "$ARCHY_LOCAL" == "1" ]] && command -v podman >/dev/null 2>&1; then
+    local orphans
+    orphans=$(podman ps -a --format '{{.Names}} {{.Status}}' 2>/dev/null \
+              | grep -iE '(^| )(stopping|removing|created)( |$)' || true)
+    if [[ -z "$orphans" ]]; then
+      record PASS "no orphaned container states" "no stopping/removing/created"
+    else
+      record FAIL "no orphaned container states" "FM9: $(echo "$orphans" | tr '\n' ';')"
+    fi
+  else
+    record WARN "no orphaned container states" "remote node — host podman sweep skipped"
+  fi
+}
+
+# ── run ────────────────────────────────────────────────────────────────────────
+echo "=============================================================="
+echo " OS-wide audit — ${BASE_URL}  ($(date '+%Y-%m-%d %H:%M:%S'))"
+echo " local=${ARCHY_LOCAL}"
+echo "=============================================================="
+section_a
+# Only proceed to apps/FM-guards if the node itself answered.
+if (( FAIL == 0 )) || [[ -n "$SESSION" ]]; then
+  section_b
+  section_c
+fi
+
+echo
+echo "=============================================================="
+echo " SCORECARD: ${PASS} pass / ${FAIL} fail / ${WARN} warn"
+echo "=============================================================="
+printf '%s\n' "${RESULTS[@]}"
+echo
+if (( FAIL > 0 )); then
+  echo "RESULT: FAIL ($FAIL critical checks failed)"
+  exit 1
+fi
+echo "RESULT: PASS"
+exit 0

tests/release/run.sh

@@ -85,14 +85,17 @@ fi
 stage "cargo-check"      timeout 580 cargo check --manifest-path core/Cargo.toml -p archipelago
 # Focused suites for the subsystems this release train touched:
 #   update::   — OTA download/apply/rollback/probe (v1.7.89 hardening)
-#   lnd        — receive address + wallet readiness (v1.7.85–.89)
+#   lnd        — receive address + wallet readiness (v1.7.85–.89), incl. the
+#                structured receive-error reason-code classifier
 #   container::image_versions — image pinning / false-update detection
 #   scanner    — RAII in-flight guard (v1.7.84)
+#   drift      — published-port drift detection (the .116 self-heal)
+#   missing_secret — secret-resolution names the missing file (the .198 fix)
 # 1500s: the non-incremental test-profile compile alone takes ~9 min on the
 # .116 ThinkPad; 580s expires mid-compile (exit 124) before a single test runs.
 stage "cargo-test-weekly" timeout 1500 env CARGO_INCREMENTAL=0 \
   cargo test --manifest-path core/Cargo.toml -p archipelago -- \
-  update:: lnd container::image_versions scanner
+  update:: lnd container::image_versions scanner drift missing_secret
 
 # ── Stage 4: live node smoke ─────────────────────────────────────────
 if [[ $LIVE -eq 1 ]]; then