run-gate.sh 5×-green on .228, 0 not-ok (gate-5x5.log). Records the
milestone in the header/banner, §4 workstream E, §6 sequence, and §8b;
demotes the priority banner per §6 item 6. Next: bundled testing deploy
(.116/.198 + UX frontend), multinode pass, workstreams B/C/D.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- InAppBrowser now has a bottom control bar (back/forward/reload/open-in-browser/
close) mirroring the web mobile footer, plus a centered loading screen
(app favicon + progress bar) instead of a bare top bar over black.
- Commit a repo-dedicated debug keystore and pin signingConfigs.debug to it so
every machine — and the published companion download — signs debug builds with
the SAME key (fixes "App not installed" signature-mismatch on update). Force v1+v2.
- Bump versionCode 10→11, versionName 0.4.6→0.4.7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Mobile launches use the store-driven panel (no route push) so the background
tab no longer changes and closing returns to where you launched from.
- Tab-only apps open directly (in-app WebView on companion / new tab on PWA) —
no "this app opens in a tab" interstitial.
- Shared AppLoadingScreen (app icon + progress bar) on the app session and the
legacy iframe overlay instead of a black screen.
- Pin the dashboard to 100dvh on mobile so the mesh chat/tools panes stop sliding
under the bottom tab bar in mobile browsers (no-op in the companion WebView).
- ElectrumX/electrs/electrs-ui ids now resolve to the real ElectrumX icon in My Apps.
- isMobile made reactive so overlay/footer/teleport decisions track the viewport.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5× run #4 flaked iter4 on "immich exposes its web UI lan-address
(port 2283)": container-list returned lan_address=null because
immich_server was momentarily mid-recreate when the read-only tier
queried it (passed the other 4 iterations; immich_server does publish
0.0.0.0:2283->2283). Same single-shot-read class as the bitcoin-knots
state probe — poll <=30s for the exposed port instead of one read. A
genuinely unexposed immich never publishes 2283, so real port drift
is still caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record the overnight 5× outcome (2/5) and the triage: all three
fails were distinct one-offs. iter1 #5 bitcoin-knots = pre-launch
churn (hardened anyway); iter2 #74 + iter5 #73 = one real
orchestrator bug (phantom stack-member injection in
ordered_containers_for_start), now fixed + live-verified on .228.
Update the resume check command to gate-5x4.log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
package.restart resolved its container list via
ordered_containers_for_start, which injected every name from the
union startup_order list that wasn't already present — including
variant names not live on a given node (mysql-mempool,
archy-mempool-api, archy-mempool-web). The phantom mysql-mempool is
2nd in the mempool start order, so do_orchestrator_package_start hit
its unknown-app-id fallback, do_package_start failed the inspect
("no such object"), and the `?` aborted the whole start sequence —
leaving mempool-api + the frontend down until the health monitor
recovered them minutes later. That was the source of the 5× gate
flakes #73 (frontend not running in 180s) and #74 (api not queryable
in 300s); root-caused from the .228 journal
("Start failed: mysql-mempool").
Replace the inject-then-sort logic with a pure helper
order_present_containers that orders only the actually-present
containers and never adds phantom entries. startup_order remains a
union of name variants across install generations — it's now used
purely to order what's live, not to inject what isn't. +3 unit tests.
Also harden bitcoin-knots.bats "valid state" probe: poll ≤30s for a
settled state instead of a single-shot read, so a container caught
mid-reconcile (transient restarting/configured) can't flake a 20-min
iteration. A genuinely-stuck container never settles, so real
breakage is still caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename run-20x.sh → run-gate.sh, default ARCHY_ITERATIONS 20→5, and scrub
20× references across CLAUDE.md, the master plan, TESTING.md, app-registry
status, the orchestrator/config doc-comments, and the bats suites. Also add
a minimal fail() helper to mempool.bats so guard failures report cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The frontend nginx used a literal proxy_pass host with no resolver, so it
pinned mempool-api's IP at worker startup. When the backend restarts (gate,
OTA, crash, reboot re-IPAM) podman reassigns its IP and nginx keeps proxying
to the dead one -> /api hangs, websocket 502s, UI shows 'offline' until a
manual nginx reload. Same stale-upstream-IP class as the netbird 502.
Fix: mempool-frontend:v3.0.1 rewrites the generated nginx-mempool.conf to
re-resolve the backend per-request via 'resolver' + a variable proxy_pass.
Resolver address is read from /etc/resolv.conf (podman aardvark-dns answers
on the network gateway, not Docker's 127.0.0.11). Per-location path mapping
preserved (ws -> '/', /api/v1 identity via no-URI, /api/ -> /api/v1/ rewrite).
Proven on .228: backend IP change now auto-recovers with no reload; the
literal-host control still 502s. Migrated the manifest off the retired
tx1138 registry to vps2.
Also: mempool.bats #74 waited only 180s post-restart (the slow path) and
called an undefined 'fail' helper (status 127). Bumped to 300s to match the
passing parity probes and emit a real failure instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures: .228 1x-GREEN (110/110); hardened 5x DETACHED on .228 (/tmp/gate-5x2.log,
nohup — survives terminal close) with the exact check-from-any-machine command; all
shipped code fixes (commits) + deploy state (.228 + .198); node-state fixes NOT in
repo (lnd nginx proxy 8081->18083, home-assistant orphan unit removed, electrumx
re-registered); the run-ON-the-node lesson; and remaining work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1x gate is green; the 5x failed iters 1-2 on readiness-under-churn (apps DO
recover — lnd synced, mempool just mid-restart when probed — but slower than the
windows when restarted back-to-back). Hardening:
- run-20x.sh: best-effort settle_stack() before each iteration (wait for
mempool-api/frontend + lnd RPC healthy, 180s, on-node, never fails the run).
- required containers present/running (80/81): wait-loops (180s) not single-shot.
- mempool api/frontend (87/88): retry ~180s not single-shot.
- mempool queryable (74): 60s->180s. lnd restart-running (64): 120s->240s.
lnd getinfo (60): 90s->240s retry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Companion app: open every app in the in-app WebView (not just non-iframeable),
carrying the mobile-iframe footer controls into the WebView. Mobile web (PWA):
open tab-apps directly in a new tab. No interstitial on either surface. Touch
points + prior commits (b5a9deb8, d1fbcd9b) noted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per direction: the gate is now 5x green ON .228 only (run on the node, not via RPC).
Fleet/multinode verification (.198 + others) moved to a new docs/multinode-testing-plan.md
with the bootstrap recipe, per-node preconditions (synced archival bitcoin, no stale
nginx proxy targets, no orphan quadlet units), node roster, and cross-node suites.
Updated CLAUDE.md, master-plan SS5/SS6/SS8b/WS-E, and TESTING.md release gates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Curated files loader now MERGES per top-level folder: dropping real files into
demo/files/Music/ swaps only Music and keeps the sample Documents/Photos/Videos
(verified). Media plays with the Range support already in place.
- AIUI index.html: a ?seed bootstrap pre-loads the example "Content Showcase"
conversation into AIUI's IndexedDB by calling the bundle's own
seedPromptsToConversation() (identical to its /seed command), so the chat
history isn't empty when the demo points users to "previous chats". Guarded by
try/catch + an existence check; no-op without ?seed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- IndeeHub + Mempool: nginx reverse-proxy + strip X-Frame-Options/CSP + sub_filter
rewrite of absolute asset paths so the frame-busting SPAs load in the iframe
(mempool.space remains best-effort — third-party CSP/ws may still limit it).
- AIUI iframe gets ?mockArchy in demo → its built-in mock node data loads.
- Pay-with-mobile QR: invoice settles after ~2s (backend gate keyed by
payment_hash) and the poll tightened to 1s, so the QR is visible before auto-pay.
- Wallet settings: dummy Cashu mints (4) + Fedimint federations (2, 222,500 sats),
interactive per session (streaming.list/configure-mints, wallet.fedimint-list/
join/balance).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a proper on-node .228 run (synced bitcoin, 4-fix binary) the lifecycle matrix is
green; these 4 were test-harness issues:
- lnd 'recovers after restart' (65): bump retry window 90s->240s. lnd cold-restart
recovery (wallet unlock + bitcoind reconnect + graph sync) exceeds 90s on a loaded
node but DOES complete (synced_to_chain:true).
- bitcoin ui responds (89): retry ~120s instead of single-shot (companion nginx may
have just been recreated by the companion-survives test).
- probe_app_url (99 lnd proxy + all ui-coverage proxy probes): retry up to 90s for
post-restart proxy/UI readiness instead of single-shot.
- required endpoints after restart (94): :8081 is nginx-proxy-manager, an OPTIONAL
app (not in required_containers) — only assert it when NPM is installed; and make
the trailing lncli getinfo a retry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The catalog pointed at a non-existent nostr.svg (handleImageError only falls
back .png→.svg, so an .svg miss stays broken). Point it at the existing nostr
icon. fedimint icon already uses fedimint.png (exists); the stale fedimint.jpg
request is resolved by /api/app-catalog now serving the local catalog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lnd's RPC isn't ready until its wallet auto-unlocks on (re)start, which lags the
container 'running' state — single-shot lncli getinfo raced that window and
false-failed (gate tests 60 + 85). Retry up to ~90s like a health probe. lnd is
functional (getinfo returns cleanly once ready).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Mempool and IndeeHub load their real site directly in the iframe (reverted the
proxy/new-tab — per request "use https://indee.tx1138.com/").
- Real app UIs now served as whole static dirs under /app/<id>/ (express.static)
so their bundled assets (qrcode.js, css, bg images) resolve; /app/<id>/assets/*
redirect to the frontend's shared assets. Fixes the console 404 cascade.
- Bitcoin Core/Knots: register rpc/v1 + bitcoin-rpc on their paths (relay-status
no longer 404s); per-impl bitcoin-status preserved.
- AIUI chat returns a fixed line in demo ("Not available in demo, check out the
previous chats to experience AIUI") instead of calling Claude — no key spend.
- Add /api/app-catalog (serves the baked catalog) to stop that 404.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- App UIs now use the real registry shells with dummy data: bitcoin-ui for
Bitcoin Core (Satoshi subversion) and Bitcoin Knots (Knots subversion) via
per-path /app/bitcoin-{core,knots}/bitcoin-status; the real lnd-ui (mock
/proxy/lnd/v1/getinfo+channels, /lnd-connect-info, /api/container/logs); the
static fedimint-ui. ElectrumX already on the real electrs-ui. Custom mock UIs
dropped — accurate UX.
- IndeeHub loads in the iframe: nginx reverse-proxies /app/indeedhub/ →
indee.tx1138.com and strips X-Frame-Options/CSP (it blocked framing before).
- Mempool opens in a new tab (mempool.space can't be iframed).
- Cloud media playback: HTTP Range support in the curated-file server so audio/
video can stream and seek (needs real files dropped into demo/files/).
- Dockerfile/.dockerignore copy docker/lnd-ui + docker/fedimint-ui.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- immich restart: bump wait 120s->240s. Restart = ordered stop+start of the 3-
container stack (postgres->redis->server w/ DB migrations), so it needs at least
as long as the start test (180s) — the old 120s was inconsistent and false-failed
on loaded nodes. immich does return to running.
- fedimint orphan check: the unanchored 'total' regex (^fedimint) counts the
legitimate fedimint-clientd (dual-ecash bridge) but the anchored 'known' regex
omitted it -> total>known false orphan on every node running fedimint-clientd.
Add fedimint-clientd to known.
Both run as LOCAL podman/systemctl on the gate runner, so they test the runner node
(.116), not the RPC target — surfaced while driving the .228 gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Mock app UIs (ElectrumX, LND, Fedimint, Bitcoin Core) + the "Not available"
notice now use the Archipelago black theme and show the app's My-Apps icon.
- Bitcoin Core gets its own UI (/app/bitcoin-core/) so it no longer shows Bitcoin
Knots branding; the Knots-branded bitcoin-ui shell is reserved for Bitcoin Knots.
- ElectrumX now serves the real electrs-ui shell (+ qrcode.js + a dummy
/electrs-status) with the correct ElectrumX icon; "Electrs" renamed to ElectrumX.
- My Apps: pre-install Bitcoin Knots again, drop ThunderHub, rename Electrs→ElectrumX.
- App store no longer shows "Checking…" forever in demo — non-demoable apps show
"No demo" immediately (skip the container-scan state).
- Relay endpoint no longer reveals a real domain (randomised host).
- Dockerfile/.dockerignore copy docker/electrs-ui into the backend image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Independent companion loop (452f05d8) validated on .228: deleted archy-electrs-ui
recreates in ~10s (was stuck 100s+). Also: companion-survives bats does LOCAL
rm/systemctl --user, so running it from .116 via RPC tests .116's companions with
.116's binary, NOT the remote target — must run ON the target node. Explains the
'failed on both nodes' runs (both silently tested .116).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The companion-unit repair stage ran at the END of each boot-reconciler tick, after
reconcile_existing(). On a heavily loaded node that per-app pass takes >60-90s, so a
deleted/lost companion unit (electrs-ui, bitcoin-ui, …) wasn't repaired within any
reasonable window (gate test 31 'deleted unit recreated within one reconcile tick'
timed out at 90s on the 45-app .228 node). Detecting + rewriting a companion unit is
cheap, so spawn it as its own ~interval(30s) loop, independent of the slow app pass.
Handle is aborted when the main loop exits (shutdown uses notify_one, so a second
waiter would steal the wake permit). tick() is now app-reconcile only.
All 4 boot_reconciler cadence tests still green (companion_stage=false in tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- nginx-demo.conf + vite proxy now route every /app/<id>/ to the mock backend, so
the per-app mock UIs and the generic "Not available in the demo" notice render
(previously only /app/filebrowser was proxied → most apps 404'd).
- Mempool and IndeeHub now load in the in-app iframe (not a new tab).
- Add an LND Lightning mock UI (channels, balances, routing) with dummy data;
lnd/thunderhub are demoable. Notice page reworded to "Not available in the demo".
- Fix missing icons: Bitcoin Core → bitcoin-core.png, Mempool → mempool.webp.
- Pre-install only Bitcoin Core (drop duplicate Bitcoin Knots; still installable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Last 2 .228 stragglers confirmed load/timing, not bugs: test 31 (companion recreate)
= contamination + ~108s reconcile cadence > 90s window; test 55 (immich restart) =
heavy stack restarts >120s under load but DOES return. Path to literally-green gate
is infra (bitcoin sync, re-quadletize .228) + minor test-window tuning. Optional
product improvement noted: independent ~30s companion-reconcile cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
companion::reconcile only recreates a deleted companion unit when its parent
backend is in manifest_ids. On contaminated .228, electrumx ran as plain podman
and was NOT a tracked manifest install (manifest on disk but unloaded), so the
reconciler never iterated it -> archy-electrs-ui companion orphaned. Proven:
package.install electrumx re-registered it + restored the companion. Self-heal
logic is sound; test 31 clears on re-quadletize. electrumx on .228 de-contaminated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backend COPY of docker/bitcoin-ui failed in Portainer because .dockerignore
(* + whitelist) excluded it. Re-include docker/ then exclude its contents except
bitcoin-ui, so the build context contains the Bitcoin UI mock shell. demo/files is
already covered by !demo/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- demo/files/<Folder>/<file> becomes the cloud's content for every visitor
(read-only; "private login" = git/repo access). Text inlined, binaries streamed
from disk; empty folder falls back to the built-in seeded set.
- Dockerfile.backend now copies docker/bitcoin-ui and demo/files into the image
(they live outside neode-ui/) — this also fixes the Bitcoin UI mock, which the
backend reads from /docker/bitcoin-ui and was previously absent in the container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.228 104/110, .198 94/110 with the 3-fix binary. Every package.stop test passes on
healthy apps. .198's 14/16 failures trace to bitcoin in IBD (test 83: ~137k blocks
behind) cascading to lnd/btcpay/electrumx/mempool. 2 node-independent: companion
recreate (31, both nodes), fedimint orphan pollution (44). Path to green 5x gate is
now infra (sync bitcoin, re-quadletize .228) + minor (test 31), not lifecycle bugs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Claude proxy injects a system-prompt describing this node (version, signet
chain + height, wallet balances, installed apps, 5 FIPS peers / 12 trusted nodes)
into every demo chat request. The assistant answers local-node and Bitcoin
questions with the node's real-looking data automatically — no /seed needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fips.status reports installed+active with 5 authenticated peers and an anchor
connection; list/add/remove/apply seed-anchors and reconnect/install all resolve
to working states so the FIPS Mesh + Seed Anchors cards light green in the demo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Tx/explorer links open mempool.space/testnet/tx/<id>; the backend hydrates the
wallet's transactions with REAL recent testnet txids at startup (best-effort,
falls back to mock hashes offline). Mempool app + demo-external apps open in a
new tab; deep-link paths are carried through.
- Add the content.* paid-download handlers the buy flow needs (owned-list,
preview-peer, download-peer-{paid,invoice,onchain}, request-invoice,
invoice-status, request-onchain, onchain-status) — every path resolves to a
success state with testnet receive addresses / bolt11 invoices so visitors can
walk the full buy → unlock journey.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
App launching (DEMO):
- resolveAppUrl routes every app to its demo target: mock UIs for Bitcoin Core,
ElectrumX, Fedimint (served by the backend), IndeeHub → iframe indee.tx1138.com,
Mempool → mempool.space/testnet (new tab); all others → a generic "Demo preview"
notice page.
- Non-demoable apps show a disabled "No demo" install button (marketplace details,
app grid, featured apps).
Onboarding:
- Demo treats the visitor as fully set up so the onboarding WIZARD (seed/identity)
is never forced; the welcome intro still replays per day. Intro CTA goes straight
to login; wizard entry points + login restart-onboarding link hidden in demo.
Network:
- federation.list-nodes now returns 12 trusted/federated nodes (9 trusted, 3
observer); transport.peers already at 5.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stop failure was 3 real product bugs (grace / reconcile-resurrection /
container-list user-stopped state), all fixed (2dad64b2, 760a32bc, 6e49ce6f) +
deployed. electrumx lifecycle suite 10/10 green (66s). fedimint 'crash loop' was
probe-induced churn (stable when left alone). Validating breadth next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turn the mock backend + UI into a public, click-to-play demo deployable as a
Portainer stack, gated behind DEMO=1 (classic single-user mock unchanged when off).
Backend (neode-ui/mock-backend.js):
- Per-session state isolation via AsyncLocalStorage + Proxy: every visitor gets
an isolated, deep-cloned copy of mockData/walletState/userState/etc., keyed by
a demo_sid cookie. Per-session WebSocket fan-out, idle reaper, session cap.
- Real per-session file storage (upload/folder/rename/delete) with a 50MB quota,
replacing the no-op filebrowser handlers; adds the missing app.filebrowser-token RPC.
- Force simulation mode (never touch a host Docker/Podman socket).
- Testnet (signet) flavor; shared login password "entertoexit".
- Report the real app version suffixed with -demo.
Frontend:
- VITE_DEMO build flag (useDemoIntro.ts): replay the intro once per calendar day
per browser; prefill + show the "entertoexit" login hint.
Deploy:
- docker-compose.demo.yml wired for DEMO, UI on :2100 (build-from-repo).
- demo-deploy/ thin stack (prebuilt :demo image refs + .env.example + README).
- .github/workflows/demo-images.yml builds/pushes archy-demo-{web,backend} images.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A user-stopped backend (electrumx, bitcoin, lnd, fedimint) kept reading 'running'
in container-list because its UI companion (electrs-ui, …) still serves the launch
port, and the state-refresh upgrades any reachable launch port to 'running'. The
gate's wait_for_container_status <app> stopped therefore never saw 'stopped'.
Fix: load the user_stopped marker in handle_container_list and force 'stopped' for
those apps before the launch-port refresh. The reconcile guard keeps the backend
down, so the marker is authoritative. package.start clears it first, so a started
app reports 'running' normally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
package.stop a dependency (e.g. electrumx, a mempool dep) and the reconciler
restarts it within ~8s: the reconcile filter's dependency_required override
re-includes a user-stopped app that an active app depends on, and the in-memory
disabled set is wiped on manifest reload — so ensure_running runs, the stopped
app's unreachable ports look like a fault, the host-port repair restarts it, and
package.stop never sticks (gate 'transitions to stopped' times out).
Fix: guard ensure_running_with_mode on the on-disk user_stopped marker (the single
choke point every reconcile flows through) → Left('user-stopped'). Explicit
install/start clear the marker first (added clear_user_stopped to orchestrator
install/start, symmetric with disabled.remove; start/restart RPC already cleared
it) so user actions are unaffected. The container itself already stopped correctly
— this stops the resurrection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix deployed to .198+.228, vaultwarden stops clean (no regression). But validation
showed the gate failures are multi-caused: (2) fedimint crash-looping/unhealthy on
both nodes can't be stopped; (3) host-listener repair watchdog restarts
port-unreachable containers fighting stop; (4) gate waits for 'stopped' but apps end
'exited'/'absent' (Exited->Stopped conversion key mismatch); (5) grace vs 60s
gate-timeout (electrumx 300s); (6) .228 contamination. Documented + re-sequenced
NEXT STEPS (fedimint health is the new top blocker).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reproduced live on CLEAN .198: package.stop fedimint -> 'podman stop -t 30
timed out after 30s' -> stop fails -> state reverts to running. Real fleet-wide
bug (NOT .228 contamination). stop_timeout_secs() per-app grace (bitcoin 600/lnd
330/electrumx 300/fedimint 60) is used by legacy stop paths but NOT the
orchestrator path: ContainerRuntime::stop_container hardcodes API ?t=10 / CLI
-t 30, and PODMAN_CLI_DEFAULT_TIMEOUT=30s == the -t grace so the await fires as
podman SIGKILLs. Fix = thread per-app grace + widen wrapper deadline; owner picks
table-based vs manifest-driven stop_grace_secs. Re-escalated to blocker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
.198 ground truth: backend apps ARE quadlet (.container files present) -> quadlet
is the intended runtime. .228's plain-podman state traced to my cascade-gate
uninstall + package.start restore (no quadlet regen). Two real robustness sub-bugs
remain (start should regen quadlet; stop podman-fallback gap). Next: canonical
gate on CLEAN .198 first to tell real-bug from contamination.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5x gate run surfaced a real blocker: package.stop does not stop electrumx/
bitcoin-knots/btcpay/fedimint/immich (container stays running; gate stop-wait
times out). Root cause chain: these backend apps run as plain podman
--restart=unless-stopped, NOT quadlet units (PODMAN_SYSTEMD_UNIT empty; only UI
companions + home-assistant have .container files; bitcoin-core.container is
.disabled). orchestrator.stop() podman-fallback fires for filebrowser but not
electrumx -> suspect loaded()/is_unknown_app_id_error gap. stop->stopped state
reporting itself is correct (filebrowser proof, user_stopped guard).
Also: corrected the canonical gate invocation (DESTRUCTIVE only, not CASCADE);
restored .228 after my cascade-gate left apps stranded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the loaded .198 the frontend churned (created → "unhealthy" → reconciler
recreates → loop). The http health check fetched / through nginx (SPA +
sub_filter) and false-failed under node load; the reconciler then treated the
frontend as wedged and recreated it. nginx binds 7777 at startup, so a tcp
liveness check passes immediately and stays green under load while still
catching a real "nginx not listening" failure. Generous retries/start_period.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live on .228 the post_install `exec` steps failed with "crun: write
cgroup.procs: Permission denied / OCI permission denied": a `podman exec`
launched from archipelago.service can't place its child in the container's
cgroup (under the service's own slice). Wrap `exec` in
`systemd-run --user --scope --quiet --collect podman exec …` so it gets its own
delegated cgroup — same trick as `podman_user_scope` for pasta starts.
`copy_from_host` (a host-side `cp`, no in-container process) stays direct.
Without this only copy_from_host worked; indeedhub happened to be unaffected
(its image pre-bakes the nginx config so the exec steps were no-ops), but the
hook capability is only generally useful with exec working. hooks unit tests
pass; live verify on .228 next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
