The prior fix's loop `container_installed "$c" && echo "$c"` makes the
function's own exit status the exit status of its LAST array entry. If
that entry isn't installed on this node (e.g. required-stack-destructive's
array ends with mempool-api, absent on .5), the whole function reports
failure even though earlier entries matched fine — and under bats' set -e,
`targets="$(installed_required_containers)"` then aborts the test outright.
required-stack.bats got lucky (its array happens to end with an installed
container) but has the identical latent bug. Caught live on .5's iteration
3 of the multinode-pass gate run. Add explicit `return 0`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live during the .5 multinode-pass run: this suite was hardcoded to
.116's exact app bundle (including the mempool stack), so any node missing
an app hard-failed instead of skipping — and a missing local fail() helper
(present in 3 sibling bats files, absent here) masked the real error as
"command not found" (exit 127). Add the same skip-if-absent idiom already
used in mempool.bats per-app, and define fail() locally like the others.
Verified: skips cleanly on .116 (no bitcoin-knots here), still exercises
real checks for apps that are installed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The 1x gate is green; the 5x failed iters 1-2 on readiness-under-churn (apps DO
recover — lnd synced, mempool just mid-restart when probed — but slower than the
windows when restarted back-to-back). Hardening:
- run-20x.sh: best-effort settle_stack() before each iteration (wait for
mempool-api/frontend + lnd RPC healthy, 180s, on-node, never fails the run).
- required containers present/running (80/81): wait-loops (180s) not single-shot.
- mempool api/frontend (87/88): retry ~180s not single-shot.
- mempool queryable (74): 60s->180s. lnd restart-running (64): 120s->240s.
lnd getinfo (60): 90s->240s retry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a proper on-node .228 run (synced bitcoin, 4-fix binary) the lifecycle matrix is
green; these 4 were test-harness issues:
- lnd 'recovers after restart' (65): bump retry window 90s->240s. lnd cold-restart
recovery (wallet unlock + bitcoind reconnect + graph sync) exceeds 90s on a loaded
node but DOES complete (synced_to_chain:true).
- bitcoin ui responds (89): retry ~120s instead of single-shot (companion nginx may
have just been recreated by the companion-survives test).
- probe_app_url (99 lnd proxy + all ui-coverage proxy probes): retry up to 90s for
post-restart proxy/UI readiness instead of single-shot.
- required endpoints after restart (94): :8081 is nginx-proxy-manager, an OPTIONAL
app (not in required_containers) — only assert it when NPM is installed; and make
the trailing lncli getinfo a retry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lnd's RPC isn't ready until its wallet auto-unlocks on (re)start, which lags the
container 'running' state — single-shot lncli getinfo raced that window and
false-failed (gate tests 60 + 85). Retry up to ~90s like a health probe. lnd is
functional (getinfo returns cleanly once ready).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
