archy/docs/incident-2026-09-01.md

Incident + follow-up tracker — 2026-09-01 (post-HTTPS-work, post-LND-0.21.2 breakage)

Live incident spanning framework-pt and shorty-s after the HTTPS/launcher work and the LND 0.18.4→0.21.2 pin bump. Root causes found on real nodes; status updated as work lands. Each fix ships with a regression test so the same class cannot silently return.

A. Root causes (all verified live)

#	Symptom	Root cause
A1	LND sends fail "Payment failed: Not Found"	LND 0.21 removed the deprecated /v1/channels/transactions REST route; backend still called it. Receive was fine; the "Failed to fetch" on framework-pt was A3 masking it.
A2	Shorty NPM restart-loops (counter 3176)	Manifest conversion (fc68c5b6) dropped (a) the /etc/letsencrypt mount NPM's s6 boot demands, and (b) NET_BIND_SERVICE — its internal nginx binds 80/443/81 and the orchestrator runs --cap-drop=ALL.
A3	framework-pt: every /rpc/v1 fetch CORS-blocked, "Failed to fetch", dashboard "not responding", mempool/indeehub frames broken	nginx sent Strict-Transport-Security: max-age=31536000; includeSubDomains on HTTPS; browsers cached it, then silently upgraded the still-open http dashboard's fetches/frames to https → scheme change = cross-origin → CORS block. HTTP is a supported mode on purpose (self-signed cert, /ca.crt flow).
A4	Mempool/IndeeHub/bitcoin-UI frames stay http:// on HTTPS pages (mixed content, "does not connect")	portAuth() looked the launch port up under the launch alias (mempool-web, lnd, bitcoin-knots…); the signed catalog declares those ports under the manifest id that owns them (archy-mempool-web, lnd-ui, bitcoin-ui) → miss → launcher fell back to http. Cache also only warmed in Store/Discover views.
A5	IndeeHub nostr sign-in dead over HTTPS	NIP-07 bridge compared event.origin for strict equality with the stored (http) app URL and replied to the stored URL as postMessage targetOrigin — both break when the frame was scheme-upgraded.
A6	Portainer "disappeared" after restart/update, then demands a setup token "see server logs"	Update to 2.45.0 recreated the container; on a fresh DB Portainer ≥2.21 mints a one-time setup token printed ONLY in container logs — hostile appliance UX. The "disappearance" was the recreate + this unknown-token first screen.

B. Fixes (code)

Fix	Files	Status
B1 LND pay via Router.SendPaymentV2 (/v2/router/send), pending-status + actionable failure reasons preserved	core/archipelago/src/api/rpc/lnd/payments.rs (+ unit tests)	✅ code
B2 Portainer setup token surfaced in the existing credentials interstitial (package.credentials → AppSidebar card with copy)	core/archipelago/src/api/rpc/package/install.rs (+ unit tests)	✅ code
B3 HSTS: none on :80, max-age=0 on :443 (actively clears cached policy)	image-recipe/configs/nginx-archipelago.conf	✅ code
B4 NPM manifest: /etc/letsencrypt mount + NET_BIND_SERVICE	apps/nginx-proxy-manager/manifest.yml	✅ code
B5 portAuth alias resolution + unanimous port-wide fallback	neode-ui/src/views/discover/curatedApps.ts	✅ code
B6 Catalog cache warmed at dashboard bootstrap	neode-ui/src/App.vue	✅
B7 NIP-07 bridge: host/port equality + reply to event.origin	neode-ui/src/stores/appLauncher.ts ✅ · neode-ui/src/views/appSession/useNostrBridge.ts ✅	✅
B8 Stale LND 0.18.4 refs in test expectations	tests/lifecycle/remote-lifecycle.sh	✅

C. Regression tests ("never again")

Test	Guards	Status
C1 Rust: router v2 response shape, nested errors, failure reasons	B1	✅
C2 Rust: setup-token log extraction (live-captured 2.45.0 line shape)	B2	✅
C3 bats: lnd-api-compat — POST /v2/router/send on the running LND must answer (never 404)	B1 vs image skew at gate time	✅ (route probe verified live on shorty: HTTP 500 ≠ 404)
C4 bats: nginx must NOT send HSTS on :80; :443 must send max-age=0	B3	✅
C5 neode-ui unit: portAuth alias + unanimous-scan (incl. bitcoin-knots→8334 https)	B5/B4-mixed-content	✅ (6 tests)
C6 neode-ui unit: bridge origin equality ignores scheme	B7	✅ (2 tests)

Backend suites: 34 targeted Rust tests green (payments v2 shape, setup-token extraction, lnd wallet/info regressions); middleware/dispatcher suite green; full neode-ui suite green (62 tests in the touched areas); production bundle built and verified to embed the alias fix. cargo fmt applied.

D. Deploy & live verification

Step	Status
D1 shorty NPM crash-loop stopped cleanly (user-stopped marker; public hosts keep serving via host nginx mirror)	✅ 12:52Z
D2 shorty live nginx HSTS patch + reload	✅ verified: :80 and :443 both answer max-age=0
D3 Regenerate catalog (releases/app-catalog.json + store copies)	✅ semantic diff = exactly the two NPM fixes
D4 User runs scripts/sign-catalog.sh (signer built at /tmp/archy-sign-bin)	✅ catalog signed + committed + pushed
D5 Commit + push (origin + gitea-vps2 OTA mirror)	✅ 9 commits pushed
D6 Release v1.8.9-alpha: scripts/create-release.sh 1.8.9-alpha (mnemonic) → scripts/publish-release-assets.sh 1.8.9-alpha gitea-vps2	✅ PUBLISHED (tag v1.8.9-alpha, releases/manifest.json live, backend+frontend assets verified by the script)
D7 OTA on shorty-s + framework-pt (Update button; shorty is on 1.8.8-alpha, daily check — hit Update now)	⬜ user action
D8 shorty: clear the NPM user-stopped marker + Start (or it starts via the fixed catalog)	✅ NPM LIVE-HEALED via the signed catalog: unit regenerated with both fixes, container up, admin UI HTTP 200 on :8081 (verified 15:42Z)
D9 framework-pt: Start Mempool — its containers are confirmed stopped (port 4080 refuses; gate answers on 7778/8334/50002/18083 so those apps will embed over https immediately)	⬜
D10 Post-deploy live checks: LND send+receive; mempool/IndeeHub/bitcoin-UI frames over https; NPM healthy + admin :8081 ✅; portainer token card on fresh DB; zero CORS errors	⬜ after nodes update

E. Follow-ups discovered during the incident (ride the NEXT release, v1.8.10+)

• Funding-modal honesty fix landed after the v1.8.9 tag (1464b1b2): the Lightning "no channel" modal now states the node's real state — pending channel confirming / balance on the far side / payment couldn't route / genuinely no channels. Ships in the next release; needs its own create-release run (one more mnemonic paste). The CHANGELOG entry for v1.8.10 should carry it.
• Nodes poll for OTA updates on daily_check — after publishing, tell the user to hit Update rather than wait for the next check.
• origin remote had a stale pushurl with a dead token (pushes failed); fixed to the canonical repo URL, stale ~/.git-credentials entry with an encoded port removed.

framework-pt immediate workaround until its OTA lands: open the dashboard by IP (http://192.168.x.x) instead of framework-pt.local, and/or clear the cached policy once via chrome://net-internals/#hsts → Delete domain security policies → framework-pt.local.