From 81743874a1411118e1108cacb04da730dcfac906 Mon Sep 17 00:00:00 2001 From: archipelago Date: Wed, 8 Jul 2026 19:48:30 -0400 Subject: [PATCH] fix(tests): bound the lnd getinfo probe + track the backend-recreate cascade gap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lnd.bats retry loop had no per-exec timeout โ€” a wedged lnd RPC (chain-blind after bitcoin-knots was recreated under it) hung one podman exec, and the whole gate, indefinitely (.228, 30+ min at test 75). timeout 10 per attempt keeps the intended ~4min bound. Tracker: new ยงC item for the root cause โ€” apps cache their backend container's IP; a backend recreate must cascade a dependent restart/alert or lnd goes chain-blind silently. Co-Authored-By: Claude Fable 5 --- docs/1.8.0-RELEASE-HARDENING-PLAN.md | 7 +++++++ tests/lifecycle/bats/lnd.bats | 5 ++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/1.8.0-RELEASE-HARDENING-PLAN.md b/docs/1.8.0-RELEASE-HARDENING-PLAN.md index 2b13a819..e8ae206b 100644 --- a/docs/1.8.0-RELEASE-HARDENING-PLAN.md +++ b/docs/1.8.0-RELEASE-HARDENING-PLAN.md @@ -202,6 +202,13 @@ modules; production request/boot paths are essentially panic-free. The real risk `no such object` phrasing added to all 3 `is_missing_container_error` classifiers. Follow-up (open): reconciler-side cleanup of an orphan umbrella `mempool` container when the split stack owns the frontend, so the stale-tile state can't arise at all. +- [ ] ๐ŸŸ  **Backend recreate must cascade to dependent apps** (found on .228 2026-07-08): + when bitcoin-knots was recreated mid-gate (one-time secret-env recreate) it got a new + archy-net IP; **lnd caches the resolved backend IP** and kept dialing the dead one + ("no route to host") for 30+ min โ€” chain-blind with open channels, silently (container + "running", health green). Repair was a manual lnd restart. Fix = reconciler/health + monitor restarts (or at least alerts on) apps whose declared backend container was + recreated; same class applies to electrumx/btcpay/nbxplorer โ†’ bitcoin links. --- diff --git a/tests/lifecycle/bats/lnd.bats b/tests/lifecycle/bats/lnd.bats index 6cd5fcd3..d338f1fa 100644 --- a/tests/lifecycle/bats/lnd.bats +++ b/tests/lifecycle/bats/lnd.bats @@ -53,8 +53,11 @@ teardown_file() { # lnd's RPC readiness LAGS the container "running" state: after a (re)start the # wallet must auto-unlock before lncli answers, so a single-shot getinfo races # that window and false-fails. Retry until ready (~90s), like a health probe. + # `timeout 10` per attempt: a wedged lnd RPC (e.g. chain-blind after its + # bitcoin backend was recreated under it, .228 2026-07-08) otherwise hangs + # a single exec โ€” and with it the whole suite โ€” indefinitely. run sh -lc 'for i in $(seq 1 80); do - podman exec lnd lncli \ + timeout 10 podman exec lnd lncli \ --tlscertpath /root/.lnd/tls.cert \ --macaroonpath /root/.lnd/data/chain/bitcoin/mainnet/readonly.macaroon \ --rpcserver localhost:10009 getinfo >/dev/null 2>&1 && exit 0