docs: incident tracker for the 2026-09-01 https/launcher/LND breakage + v1.8.9 notes

Root-cause table, fix inventory, regression-test inventory and the
deploy/live-verification checklist for today's fleet incident — written
as the working record while the fixes land, so the deploy + verify steps
can be checked off against real nodes rather than memory. CHANGELOG
carries the user-facing notes for the release these fixes ship in.

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## v1.8.9-alpha (2026-09-01)
+
+- **Lightning sends work again after the LND 0.21.2 update.** LND 0.21 removed the old synchronous payment route the node's backend paid through (`/v1/channels/transactions`) — every Lightning send answered the literal "Not Found" and the wallet showed "Payment failed: Not Found". The backend now pays through the supported Router.SendPaymentV2 route, keeps the same settle-then-report behaviour (a slow multi-hop payment is still tracked to completion, never falsely declared failed), and translates LND's failure reasons into plain advice. A new gate test speaks the payment route directly against the running LND, so an image/backend skew like this can never ship silently again.
+
+- **The node no longer pins HSTS — HTTP access is a supported mode, and it stays working.** The HTTPS listener used to send `Strict-Transport-Security: max-age=31536000; includeSubDomains`; browsers that visited HTTPS once cached that and then silently upgraded the still-open HTTP dashboard's calls to HTTPS, which is a scheme change — cross-origin — so every request died as "CORS blocked / Failed to fetch" while the node was perfectly healthy. The HTTPS listener now actively clears the cached policy (`max-age=0`) and port 80 sends no HSTS at all, which is deliberate: the node's certificate is optional and self-signed, and devices that haven't installed the CA must keep plain-HTTP access (that's what Settings → Node certificate is for). If your browser already cached the old policy, visiting the dashboard over HTTPS once after this update clears it; a gate test now refuses any config that reintroduces the pin.
+
+- **App frames open over HTTPS again — including the ones that "did not connect."** The launcher asked the signed catalog for each app's port policy under the name you click ("Mempool Web", "Bitcoin Knots"), but the catalog declares those ports under the manifest that owns them (the Mempool web container, Bitcoin UI). The lookup missed, the launcher handed the iframe an `http://` address, and the browser blocked it as mixed content — the app tile went blank or spun forever. Port resolution now follows launch aliases (mempool-web, bitcoin-knots/bitcoin-core, lnd, electrs and friends), falls back to a port-wide catalog scan when the id is unknown, and the catalog is warmed as soon as the dashboard loads rather than only in the App Store, so the very first app you open already knows which ports serve TLS.
+
+- **Signing in to IndeeHub with Nostr works over HTTPS.** The NIP-07 bridge compared the app frame's origin for exact equality with the recorded `http://` app URL — a frame the browser upgraded to HTTPS (or any scheme change) was silently ignored, and replies addressed to the stale origin were refused outright, so Nostr sign-in quietly did nothing. The bridge now matches host and port (scheme intentionally ignored) and always replies to the frame's real origin.
+
+- **Nginx Proxy Manager starts again.** Converting it to a platform manifest dropped two things its image needs: the `/etc/letsencrypt` mount its boot script hard-requires, and the `NET_BIND_SERVICE` capability its internal nginx needs to bind ports 80/443/81 under the orchestrator's `--cap-drop=ALL`. The result was an endless start/die loop (a node watched it restart 3,176 times). Both are declared in its manifest now, its certs live on unchanged under the same persistent app directory, and the signed catalog carries the fix so installed nodes heal on the next update.
+
+- **Portainer's first-run token is in the app page, not buried in "server logs."** New Portainer versions mint a one-time setup token on a fresh install and print it only to the container logs — on an appliance that meant telling the user to go read a server log to get into their own app. The token now appears in the same launch interstitial as app login credentials (with a copy button), only while first-run setup is actually pending; once the admin account exists the card disappears on its own.
+
 ## v1.8.8-alpha (2026-09-01)
 
 - **SSH over the mesh is now a first-class setting.** Settings gains an "SSH over mesh" card: off by default, and when you allow it the node's mesh firewall opens port 22 — either to every mesh peer (behind an explicit "I understand" confirmation, because that's a real exposure) or only to the mesh addresses you list. The rule is owned by the node (the `90-ssh.nft` drop-in), so it survives upgrades and daemon reinstalls, and the card tells you up front whether sshd is running, whether it listens on IPv6 (the mesh is IPv6-only — this is what a broken attempt looks like before it happens), and whether password login is on (keys-only is the recommended pairing). From Termux on your phone, `fipssh <user>@<node-npub>` connects once the toggle is on — the npub is the durable address, and the command is shown with a copy button on the card.

docs/incident-2026-09-01.md

@@ -0,0 +1,58 @@
+# Incident + follow-up tracker — 2026-09-01 (post-HTTPS-work, post-LND-0.21.2 breakage)
+
+Live incident spanning framework-pt and shorty-s after the HTTPS/launcher
+work and the LND 0.18.4→0.21.2 pin bump. Root causes found on real nodes;
+status updated as work lands. Each fix ships with a regression test so the
+same class cannot silently return.
+
+## A. Root causes (all verified live)
+
+| # | Symptom | Root cause |
+|---|---------|-----------|
+| A1 | LND sends fail "Payment failed: Not Found" | LND 0.21 **removed** the deprecated `/v1/channels/transactions` REST route; backend still called it. Receive was fine; the "Failed to fetch" on framework-pt was A3 masking it. |
+| A2 | Shorty NPM restart-loops (counter 3176) | Manifest conversion (fc68c5b6) dropped (a) the `/etc/letsencrypt` mount NPM's s6 boot demands, and (b) `NET_BIND_SERVICE` — its internal nginx binds 80/443/81 and the orchestrator runs `--cap-drop=ALL`. |
+| A3 | framework-pt: every `/rpc/v1` fetch CORS-blocked, "Failed to fetch", dashboard "not responding", mempool/indeehub frames broken | nginx sent `Strict-Transport-Security: max-age=31536000; includeSubDomains` on **HTTPS**; browsers cached it, then silently upgraded the still-open **http** dashboard's fetches/frames to https → scheme change = cross-origin → CORS block. HTTP is a supported mode on purpose (self-signed cert, /ca.crt flow). |
+| A4 | Mempool/IndeeHub/bitcoin-UI frames stay `http://` on HTTPS pages (mixed content, "does not connect") | `portAuth()` looked the launch port up under the launch alias (`mempool-web`, `lnd`, `bitcoin-knots`…); the signed catalog declares those ports under the manifest id that owns them (`archy-mempool-web`, `lnd-ui`, `bitcoin-ui`) → miss → launcher fell back to http. Cache also only warmed in Store/Discover views. |
+| A5 | IndeeHub nostr sign-in dead over HTTPS | NIP-07 bridge compared `event.origin` for strict equality with the stored (http) app URL and replied to the **stored** URL as postMessage targetOrigin — both break when the frame was scheme-upgraded. |
+| A6 | Portainer "disappeared" after restart/update, then demands a setup token "see server logs" | Update to 2.45.0 recreated the container; on a fresh DB Portainer ≥2.21 mints a one-time setup token printed ONLY in container logs — hostile appliance UX. The "disappearance" was the recreate + this unknown-token first screen. |
+
+## B. Fixes (code)
+
+| Fix | Files | Status |
+|-----|-------|--------|
+| B1 LND pay via `Router.SendPaymentV2` (`/v2/router/send`), pending-status + actionable failure reasons preserved | `core/archipelago/src/api/rpc/lnd/payments.rs` (+ unit tests) | ✅ code |
+| B2 Portainer setup token surfaced in the existing credentials interstitial (`package.credentials` → AppSidebar card with copy) | `core/archipelago/src/api/rpc/package/install.rs` (+ unit tests) | ✅ code |
+| B3 HSTS: none on :80, `max-age=0` on :443 (actively clears cached policy) | `image-recipe/configs/nginx-archipelago.conf` | ✅ code |
+| B4 NPM manifest: `/etc/letsencrypt` mount + `NET_BIND_SERVICE` | `apps/nginx-proxy-manager/manifest.yml` | ✅ code |
+| B5 `portAuth` alias resolution + unanimous port-wide fallback | `neode-ui/src/views/discover/curatedApps.ts` | ✅ code |
+| B6 Catalog cache warmed at dashboard bootstrap | `neode-ui/src/App.vue` | ⬜ |
+| B7 NIP-07 bridge: host/port equality + reply to `event.origin` | `neode-ui/src/stores/appLauncher.ts` ✅ · `neode-ui/src/views/appSession/useNostrBridge.ts` ⬜ | 🔶 |
+| B8 Stale LND 0.18.4 refs in test expectations | `tests/lifecycle/remote-lifecycle.sh` + fixtures | ⬜ |
+
+## C. Regression tests ("never again")
+
+| Test | Guards | Status |
+|------|-------|--------|
+| C1 Rust: router v2 response shape, nested errors, failure reasons | B1 | ✅ |
+| C2 Rust: setup-token log extraction (live-captured 2.45.0 line shape) | B2 | ✅ |
+| C3 bats: `lnd-api-compat` — POST `/v2/router/send` on the running LND must answer (never 404) | B1 vs image skew at gate time | ⬜ |
+| C4 bats: nginx must NOT send HSTS on :80; :443 must send `max-age=0` | B3 | ⬜ |
+| C5 neode-ui unit: portAuth alias + unanimous-scan (incl. bitcoin-knots→8334 https) | B5/B4-mixed-content | ⬜ |
+| C6 neode-ui unit: bridge origin equality ignores scheme | B7 | ⬜ |
+
+## D. Deploy & live verification
+
+| Step | Status |
+|------|--------|
+| D1 shorty NPM crash-loop stopped cleanly (user-stopped marker; public hosts keep serving via host nginx mirror) | ✅ 12:52Z |
+| D2 shorty live nginx HSTS patch + reload | ⬜ |
+| D3 Regenerate catalog (releases/app-catalog.json + store copies) | ⬜ |
+| D4 **User runs `scripts/sign-catalog.sh`** (signer build running in background) | ⬜ |
+| D5 Commit + push (origin + gitea-vps2 OTA mirror) | ⬜ |
+| D6 OTA on shorty-s + framework-pt (Update button; framework-pt has no SSH from here) | ⬜ |
+| D7 Post-deploy live checks: LND send+receive; mempool/IndeeHub/bitcoin-UI frames over https; NPM healthy + admin :8081; portainer token card on fresh DB; zero CORS errors | ⬜ |
+
+framework-pt immediate workaround until its OTA lands: open the dashboard by
+IP (`http://192.168.x.x`) instead of `framework-pt.local`, and/or clear the
+cached policy once via `chrome://net-internals/#hsts` → Delete domain security
+policies → `framework-pt.local`.