fix(mempool): resolve CORE_RPC_HOST to the actual bitcoin node (Knots/Core) (B12)

CORE_RPC_HOST was hardcoded to bitcoin-knots in three env-render paths, so on a
bitcoin-core node (container named bitcoin-core) mempool-api could not reach
Bitcoin RPC. Both node variants are reachable on archy-net by container name —
only the name differs.

- Legacy direct-podman (stacks.rs) and config.rs::get_app_config now use a new
  dependencies::detect_bitcoin_rpc_host() (pure, unit-tested pick_bitcoin_host).
- Quadlet/manifest path (the modern fleet default): add a {{BITCOIN_HOST}}
  derived-env placeholder — HostFacts.bitcoin_host + resolve_derived_env render
  it; prod_orchestrator detects Knots/Core via podman ps, resolved on demand
  only for manifests that use the placeholder. mempool-api manifest moves
  CORE_RPC_HOST from static env to derived_env: {{BITCOIN_HOST}}.

Tests: pick_bitcoin_host (5 cases incl. substring safety), container-crate
resolve_derived_env, and orchestrator mempool_core_rpc_host_follows_bitcoin_node
(core->bitcoin-core, knots->bitcoin-knots). No-regression confirmed: picker
returns bitcoin-knots live on .198. Live bitcoin-core validation pending (no
core node available). Sibling hardcodes (lnd/btcpay/electrumx/fedimint) tracked
as B12b.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

apps/mempool-api/manifest.yml

@@ -8,6 +8,12 @@ app:
     image: git.tx1138.com/lfg2025/mempool-backend:v3.0.0
     pull_policy: if-not-present
     network: archy-net
+    # CORE_RPC_HOST must follow the node's actual Bitcoin container — Knots or
+    # Core — resolved at apply time from host facts (B12). Hardcoding either
+    # breaks mempool's RPC connection on the other.
+    derived_env:
+      - key: CORE_RPC_HOST
+        template: "{{BITCOIN_HOST}}"
     secret_env:
       - key: CORE_RPC_PASSWORD
         secret_file: bitcoin-rpc-password
@@ -47,7 +53,6 @@ app:
     - ELECTRUM_HOST=electrumx
     - ELECTRUM_PORT=50001
     - ELECTRUM_TLS_ENABLED=false
-    - CORE_RPC_HOST=bitcoin-knots
     - CORE_RPC_PORT=8332
     - CORE_RPC_USERNAME=archipelago
     - DATABASE_ENABLED=true

core/archipelago/src/api/rpc/package/config.rs

@@ -752,7 +752,12 @@ pub(super) async fn get_app_config(
             None,
             None,
         ),
-        "mempool-api" => (
+        "mempool-api" => {
+            // CORE_RPC_HOST must resolve to the actual Bitcoin node container —
+            // bitcoin-knots OR bitcoin-core — else mempool-api can't reach RPC
+            // on a Core node (B12). Falls back to bitcoin-knots if undetected.
+            let bitcoin_rpc_host = super::dependencies::detect_bitcoin_rpc_host().await;
+            (
                 vec!["8999:8999".to_string()],
                 vec!["/var/lib/archipelago/mempool:/data".to_string()],
                 vec![
@@ -760,7 +765,7 @@ pub(super) async fn get_app_config(
                     "ELECTRUM_HOST=electrumx".to_string(),
                     "ELECTRUM_PORT=50001".to_string(),
                     "ELECTRUM_TLS_ENABLED=false".to_string(),
-                "CORE_RPC_HOST=bitcoin-knots".to_string(),
+                    format!("CORE_RPC_HOST={}", bitcoin_rpc_host),
                     "CORE_RPC_PORT=8332".to_string(),
                     "CORE_RPC_USERNAME=archipelago".to_string(),
                     format!("CORE_RPC_PASSWORD={}", rpc_pass),
@@ -772,7 +777,8 @@ pub(super) async fn get_app_config(
                 ],
                 None,
                 None,
-        ),
+            )
+        }
         "electrumx" | "mempool-electrs" | "electrs" => {
             (
                 vec!["50001:50001".to_string()],

core/archipelago/src/api/rpc/package/dependencies.rs

@@ -84,6 +84,78 @@ pub(super) async fn detect_running_deps() -> Result<RunningDeps> {
     })
 }
 
+/// Detect the container name of the running Bitcoin node so dependent stacks
+/// (mempool) can point CORE_RPC_HOST at the right host. Bitcoin Knots and Bitcoin
+/// Core are both reachable on archy-net by their container name — only the name
+/// differs (`bitcoin-knots` vs `bitcoin-core`), so hardcoding one breaks the
+/// other. Returns the first running BITCOIN_NAMES match; falls back to the
+/// default `bitcoin-knots` if none is detected (callers gate on has_bitcoin).
+pub(super) async fn detect_bitcoin_rpc_host() -> String {
+    let out = tokio::time::timeout(
+        std::time::Duration::from_secs(15),
+        tokio::process::Command::new("podman")
+            .args(["ps", "--format", "{{.Names}}"])
+            .output(),
+    )
+    .await;
+    if let Ok(Ok(o)) = out {
+        if o.status.success() {
+            let running = String::from_utf8_lossy(&o.stdout);
+            if let Some(name) = pick_bitcoin_host(&running) {
+                return name;
+            }
+        }
+    }
+    "bitcoin-knots".to_string()
+}
+
+/// Pure host-selection step of [`detect_bitcoin_rpc_host`], split out so it can
+/// be unit-tested without a podman runtime. Returns the first `podman ps` line
+/// whose trimmed name is one of [`BITCOIN_NAMES`]. (The Quadlet orchestrator
+/// mirrors this in `prod_orchestrator::bitcoin_host`.)
+fn pick_bitcoin_host(podman_names: &str) -> Option<String> {
+    podman_names
+        .lines()
+        .map(|l| l.trim())
+        .find(|name| BITCOIN_NAMES.contains(name))
+        .map(|name| name.to_string())
+}
+
+#[cfg(test)]
+mod bitcoin_host_tests {
+    use super::pick_bitcoin_host;
+
+    #[test]
+    fn picks_knots() {
+        let ps = "electrumx\nbitcoin-knots\narchy-mempool-db\n";
+        assert_eq!(pick_bitcoin_host(ps).as_deref(), Some("bitcoin-knots"));
+    }
+
+    #[test]
+    fn picks_core() {
+        let ps = "lnd\nbitcoin-core\nelectrumx\n";
+        assert_eq!(pick_bitcoin_host(ps).as_deref(), Some("bitcoin-core"));
+    }
+
+    #[test]
+    fn picks_plain_bitcoin() {
+        assert_eq!(pick_bitcoin_host("bitcoin\n").as_deref(), Some("bitcoin"));
+    }
+
+    #[test]
+    fn none_when_no_bitcoin_node() {
+        let ps = "electrumx\nlnd\narchy-mempool-db\n";
+        assert_eq!(pick_bitcoin_host(ps), None);
+    }
+
+    #[test]
+    fn ignores_substring_matches() {
+        // A companion UI container must NOT be mistaken for the node itself.
+        let ps = "archy-bitcoin-ui\nbitcoin-knots-foo\n";
+        assert_eq!(pick_bitcoin_host(ps), None);
+    }
+}
+
 /// Verify that required dependency services are running before installing an app.
 /// Returns an error with a user-friendly message if dependencies are missing.
 pub(super) fn check_install_deps(package_id: &str, deps: &RunningDeps) -> Result<()> {

core/archipelago/src/api/rpc/package/stacks.rs

@@ -1152,6 +1152,9 @@ impl RpcHandler {
         let deps = super::dependencies::detect_running_deps().await?;
         super::dependencies::check_install_deps("mempool", &deps)?;
         let (_, rpc_pass) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
+        // CORE_RPC_HOST must match the actual Bitcoin node container name —
+        // bitcoin-knots OR bitcoin-core — else mempool-api can't reach RPC (B12).
+        let bitcoin_rpc_host = super::dependencies::detect_bitcoin_rpc_host().await;
 
         install_log("INSTALL START: mempool (stack: mariadb + mempool-api + mempool-web)").await;
 
@@ -1275,7 +1278,7 @@ impl RpcHandler {
                 "-e",
                 "ELECTRUM_TLS_ENABLED=false",
                 "-e",
-                "CORE_RPC_HOST=bitcoin-knots",
+                &format!("CORE_RPC_HOST={}", bitcoin_rpc_host),
                 "-e",
                 "CORE_RPC_PORT=8332",
                 "-e",

core/archipelago/src/container/prod_orchestrator.rs

@@ -772,6 +772,8 @@ pub struct ProdContainerOrchestrator {
     use_quadlet_backends: bool,
     #[cfg(test)]
     test_disk_gb: Option<u64>,
+    #[cfg(test)]
+    test_bitcoin_host: Option<String>,
 }
 
 struct FileSecretsProvider {
@@ -832,6 +834,8 @@ impl ProdContainerOrchestrator {
             use_quadlet_backends: config.use_quadlet_backends,
             #[cfg(test)]
             test_disk_gb: None,
+            #[cfg(test)]
+            test_bitcoin_host: None,
         })
     }
 
@@ -850,6 +854,7 @@ impl ProdContainerOrchestrator {
             secrets_dir: PathBuf::from("/var/lib/archipelago/secrets"),
             use_quadlet_backends: false,
             test_disk_gb: None,
+            test_bitcoin_host: None,
         }
     }
 
@@ -2314,9 +2319,45 @@ impl ProdContainerOrchestrator {
             host_ip,
             host_mdns,
             disk_gb,
+            // Cheap default; resolve_dynamic_env fills the real node name on
+            // demand (it costs a podman call) only for manifests that use
+            // {{BITCOIN_HOST}}, rather than every app on every reconcile.
+            bitcoin_host: "bitcoin-knots".to_string(),
         }
     }
 
+    /// Container name of the running Bitcoin node (`bitcoin-knots` or
+    /// `bitcoin-core`) for the `{{BITCOIN_HOST}}` derived-env placeholder.
+    /// Synchronous `podman ps` to match the surrounding host-fact detection;
+    /// defaults to `bitcoin-knots` when none is running (B12).
+    fn bitcoin_host(&self) -> String {
+        #[cfg(test)]
+        if let Some(host) = &self.test_bitcoin_host {
+            return host.clone();
+        }
+        // Mirrors api::rpc::package::dependencies (the legacy install path);
+        // both Bitcoin node variants are reachable on archy-net by name.
+        const BITCOIN_NAMES: &[&str] = &["bitcoin-knots", "bitcoin-core", "bitcoin"];
+        let names = Command::new("podman")
+            .args(["ps", "--format", "{{.Names}}"])
+            .output()
+            .ok()
+            .filter(|o| o.status.success())
+            .map(|o| String::from_utf8_lossy(&o.stdout).into_owned())
+            .unwrap_or_default();
+        names
+            .lines()
+            .map(|l| l.trim())
+            .find(|name| BITCOIN_NAMES.contains(name))
+            .map(|name| name.to_string())
+            .unwrap_or_else(|| "bitcoin-knots".to_string())
+    }
+
+    #[cfg(test)]
+    pub fn set_bitcoin_host_for_test(&mut self, host: &str) {
+        self.test_bitcoin_host = Some(host.to_string());
+    }
+
     fn detect_host_ip() -> Option<String> {
         let output = Command::new("hostname").arg("-I").output().ok()?;
         if !output.status.success() {
@@ -2400,7 +2441,18 @@ impl ProdContainerOrchestrator {
     }
 
     fn resolve_dynamic_env(&self, manifest: &mut AppManifest) -> Result<()> {
-        let facts = self.detect_host_facts();
+        let mut facts = self.detect_host_facts();
+        // Only pay the podman cost to detect Knots-vs-Core when this manifest
+        // actually templates the Bitcoin node into its env (mempool — B12).
+        if manifest
+            .app
+            .container
+            .derived_env
+            .iter()
+            .any(|e| e.template.contains("{{BITCOIN_HOST}}"))
+        {
+            facts.bitcoin_host = self.bitcoin_host();
+        }
         let mut env = manifest.app.environment.clone();
         env.extend(manifest.app.container.resolve_derived_env(&facts));
 
@@ -3489,6 +3541,35 @@ app:
         assert!(!calls.iter().any(|c| c.starts_with("build_image:")));
     }
 
+    #[tokio::test]
+    async fn mempool_core_rpc_host_follows_bitcoin_node() {
+        // B12: mempool's CORE_RPC_HOST must resolve to whichever Bitcoin node
+        // container is running (Knots OR Core), not a hardcoded value.
+        let yaml = "app:\n  id: mempool-api\n  name: mempool-api\n  version: 1.0.0\n  container:\n    image: x:1\n    derived_env:\n      - key: CORE_RPC_HOST\n        template: \"{{BITCOIN_HOST}}\"\n";
+
+        for (node, expected) in [
+            ("bitcoin-core", "bitcoin-core"),
+            ("bitcoin-knots", "bitcoin-knots"),
+        ] {
+            let rt = Arc::new(MockRuntime::default());
+            let mut orch = orch_with(rt).await;
+            orch.set_bitcoin_host_for_test(node);
+
+            let mut manifest = AppManifest::parse(yaml).unwrap();
+            orch.resolve_dynamic_env(&mut manifest).unwrap();
+
+            assert!(
+                manifest
+                    .app
+                    .environment
+                    .iter()
+                    .any(|e| e == &format!("CORE_RPC_HOST={expected}")),
+                "node={node}: expected CORE_RPC_HOST={expected}, got {:?}",
+                manifest.app.environment
+            );
+        }
+    }
+
     #[tokio::test]
     async fn install_fresh_build_when_image_absent() {
         let rt = Arc::new(MockRuntime::default());

core/container/src/manifest.rs

@@ -858,6 +858,11 @@ pub struct HostFacts {
     /// `/` if the data partition is not yet mounted). Drives the
     /// prune-vs-full-node decision in bitcoin-knots custom_args.
     pub disk_gb: u64,
+    /// Container name of the running Bitcoin node — `bitcoin-knots` or
+    /// `bitcoin-core` — so dependents (mempool's CORE_RPC_HOST) reach the
+    /// right host. Both are reachable on archy-net by their container name;
+    /// only the name differs. Falls back to `bitcoin-knots` when undetected.
+    pub bitcoin_host: String,
 }
 
 impl HostFacts {
@@ -868,13 +873,14 @@ impl HostFacts {
             host_ip: "192.168.1.116".to_string(),
             host_mdns: "archi-thinkpad.local".to_string(),
             disk_gb: 2000,
+            bitcoin_host: "bitcoin-knots".to_string(),
         }
     }
 }
 
 /// Supported placeholder names in `DerivedEnv::template`. Keep in sync
 /// with `HostFacts`. Centralized so validation and rendering agree.
-const DERIVED_PLACEHOLDERS: &[&str] = &["HOST_IP", "HOST_MDNS", "DISK_GB"];
+const DERIVED_PLACEHOLDERS: &[&str] = &["HOST_IP", "HOST_MDNS", "DISK_GB", "BITCOIN_HOST"];
 
 fn validate_derived_template(key: &str, template: &str) -> Result<(), ManifestError> {
     // Walk `{{NAME}}` occurrences and ensure each NAME is recognized.
@@ -957,7 +963,8 @@ impl ContainerConfig {
                     .template
                     .replace("{{HOST_IP}}", &facts.host_ip)
                     .replace("{{HOST_MDNS}}", &facts.host_mdns)
-                    .replace("{{DISK_GB}}", &facts.disk_gb.to_string());
+                    .replace("{{DISK_GB}}", &facts.disk_gb.to_string())
+                    .replace("{{BITCOIN_HOST}}", &facts.bitcoin_host);
                 format!("{}={}", e.key, value)
             })
             .collect()
@@ -1463,6 +1470,10 @@ app:
                     key: "INFO".to_string(),
                     template: "{{HOST_IP}}-{{DISK_GB}}".to_string(),
                 },
+                DerivedEnv {
+                    key: "CORE_RPC_HOST".to_string(),
+                    template: "{{BITCOIN_HOST}}".to_string(),
+                },
             ],
             secret_env: vec![],
             data_uid: None,
@@ -1471,11 +1482,13 @@ app:
             host_ip: "192.168.1.116".to_string(),
             host_mdns: "archi-thinkpad.local".to_string(),
             disk_gb: 2000,
+            bitcoin_host: "bitcoin-core".to_string(),
         };
 
         let out = c.resolve_derived_env(&facts);
         assert_eq!(out[0], "FM_API_URL=ws://archi-thinkpad.local:8174");
         assert_eq!(out[1], "INFO=192.168.1.116-2000");
+        assert_eq!(out[2], "CORE_RPC_HOST=bitcoin-core");
     }
 
     struct MapSecretsProvider {

tests/production-quality/TRACKER.md

@@ -8,13 +8,14 @@ cd ~/Projects/archy && git fetch gitea-vps2 && git checkout main && git reset --
 ```
 Then continue from "IN PROGRESS" below.
 
-**Committed & ready for .97 (vps2 main):** B5 (LND CORS, verified .116/.198/.103), B1, B2, B4, B14, B21, B3 (incl. /api/peer-content nginx via bootstrap), B15, B7, **B13 (fedimint CSS self-heal — main conf + HTTPS snippet, verified .198 both paths app-icon 404→200)**. B6 pruned-gate already live.
+**Committed & ready for .97 (vps2 main):** B5 (LND CORS, verified .116/.198/.103), B1, B2, B4, B14, B21, B3 (incl. /api/peer-content nginx via bootstrap), B15, B7, **B13 (fedimint CSS self-heal — main conf + HTTPS snippet, verified .198 both paths app-icon 404→200)**, **B12 (mempool bitcoin-host detect across 3 render paths — unit-tested; live bitcoin-core validation pending)**. B6 pruned-gate already live. = 12 fixes.
 
-**IN PROGRESS — pick up at B12.** B13 DONE (committed this session; bootstrap.rs self-heals both the main conf and the HTTPS app-proxy snippet — see B13 entry below for full verification). REMAINING:
+**IN PROGRESS — pick up at B16.** B13 + B12 DONE (committed; see their entries below for full detail). REMAINING:
-1. **B12** (mempool host detect — stacks.rs:1278 hardcodes CORE_RPC_HOST=bitcoin-knots; fails on bitcoin-core nodes → dynamic host detect; backend, medium risk, test .116).
+1. **B16** (bitcoin status retain — needs a UI test).
-2. Then **B16** (bitcoin status retain — UI-test), **B6** no-node-present half, **B14b** (FIPS reachability depth), **B22/B23** (peer download + group chat — need live repro), B9/B10/B11/B17/B18/B19, B8 (low), B20 (mesh-headers feature).
+2. Then **B6** no-node-present half, **B12b** (sibling bitcoin-host hardcodes: LND/BTCPay/electrumx/fedimint + mempool dep declaration — reuse `{{BITCOIN_HOST}}`; needs validation, esp. LND/fedimint), **B14b** (FIPS reachability depth), **B22/B23** (peer download + group chat — need live repro), B9/B10/B11/B17/B18/B19, B8 (low), B20 (mesh-headers feature).
+3. **Loose end:** 4 pre-existing prod_orchestrator test failures (generated-files/data_uid fixtures use disallowed tempdir volume sources) — see B12 NOTE; separate small fix.
 
-Note: .198 is currently running a sideloaded .97-dev binary (md5 4c83803d, built from this B13 commit) — NOT an official release. Reflashing/OTA will replace it.
+Note: .198 is running a sideloaded B13-era .97-dev binary (md5 4c83803d). The B12 binary was built (`core/target/release/archipelago`) but NOT sideloaded (mempool isn't on .198; .198 is Knots so B12 is a no-op there). Reflashing/OTA replaces the dev binary.
 
 **Ship .97 when ready:** ./scripts/create-release.sh 1.7.97-alpha (curate CHANGELOG ≥3 layman bullets first + run scripts/sync-whats-new.py; SKIP_RELEASE_TESTS=1 only for the 2 known-flaky vitest timing tests) → scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2 → git push gitea-vps2 main + tag. (gitea-local push fails: token rejected — non-blocking.)
 
@@ -103,7 +104,16 @@ Recurring crash ("still" → prior attempts). Check container logs + resource li
 ### B11 — Companion app: "open in external browser" apps don't work — TODO
 Apps meant to open in a new/external browser don't launch from the companion app; need the phone-default-browser request-modal pattern mobile apps use. Relates to v1.7.90 "open in new tab from companion app".
 
-### B12 — Mempool not connecting — ROOT-CAUSED (stacks.rs:1278 hardcodes CORE_RPC_HOST=bitcoin-knots; fails on bitcoin-core nodes. Fix=dynamic host detect. Backend, medium risk, test .116)
+### B12 — Mempool not connecting — FIXED (mempool host detect, 3 paths; unit-tested). Live bitcoin-core validation PENDING (no core node available).
+**Bigger than the original "stacks.rs:1278" framing.** `CORE_RPC_HOST=bitcoin-knots` was hardcoded in THREE env-render paths; on a bitcoin-core node the container is named `bitcoin-core`, so mempool-api can't resolve RPC. Both Knots and Core are reachable on `archy-net` by container name — only the name differs.
+- **Path 1 — legacy direct-podman** (`stacks.rs::install_mempool_stack`, used when no orchestrator): now `format!("CORE_RPC_HOST={}", detect_bitcoin_rpc_host())`. FIXED.
+- **Path 2 — `config.rs::get_app_config`** (install.rs legacy path): same. FIXED.
+- **Path 3 — Quadlet/manifest (THE MODERN FLEET PATH, e.g. .198)**: `prod_orchestrator` renders env from `apps/mempool-api/manifest.yml` static YAML. FIXED via a new `{{BITCOIN_HOST}}` derived-env placeholder: `HostFacts.bitcoin_host` (container/manifest.rs) + `resolve_derived_env` renders it; `prod_orchestrator::bitcoin_host()` detects Knots/Core via `podman ps` (test-injectable `set_bitcoin_host_for_test`); resolved on-demand only for manifests using the placeholder (perf). mempool-api manifest moved `CORE_RPC_HOST` from static env → `derived_env: {{BITCOIN_HOST}}`.
+- New helper `dependencies::detect_bitcoin_rpc_host()` + pure `pick_bitcoin_host()`.
+- **TESTS (all green):** `pick_bitcoin_host` 5 cases (knots/core/plain/none/substring-safety); container-crate `resolve_derived_env` renders `{{BITCOIN_HOST}}`; orchestrator `mempool_core_rpc_host_follows_bitcoin_node` (core→bitcoin-core, knots→bitcoin-knots). No-regression verified: picker returns `bitcoin-knots` live on .198 (so Knots nodes unchanged; existing mempool installs see no env drift).
+- **VALIDATION GAP:** cannot exercise on a live bitcoin-core node (none available; .198 is Knots where the fix is a no-op). Need a Core node to confirm end-to-end.
+- **FOLLOW-UP (B12b, NOT done):** same hardcode exists for siblings on bitcoin-core nodes — `config.rs` lnd(:724)/btcpay(:739)/electrumx(:782), and `prod_orchestrator::resolve_dynamic_env` fedimint `FM_BITCOIND_URL=...bitcoin-knots` (~:2425). Plus mempool-api manifest `dependencies: bitcoin-knots` (line 18) is Knots-specific bookkeeping (install-time check already accepts Core via BITCOIN_NAMES, so non-blocking). All can reuse `{{BITCOIN_HOST}}`. Deferred per user (mempool-only scope) — each needs its own validation, esp. LND/fedimint.
+- **NOTE (unrelated pre-existing failures):** 4 prod_orchestrator tests fail on clean HEAD too — `install_applies_data_uid_chown_before_create`, `install_writes_manifest_generated_files_before_create`, `manifest_generated_files_{do_not_overwrite_by_default,can_overwrite_when_declared}` — their fixtures pass tempdir volume sources that `validate_bind_source` rejects (only `/var/lib/archipelago/*` + 2 sockets allowed). NOT caused by B12; worth a separate fix.
 mempool can't reach the Bitcoin backend on some nodes. Investigate on .116. Check mempool→electrs→bitcoind wiring + deps.
 
 ### B13 — Fedimint UI not applying CSS — FIXED + VERIFIED on .198 (both HTTP + HTTPS)