archy/docs/incident-2026-09-01.md

# Incident + follow-up tracker — 2026-09-01 (post-HTTPS-work, post-LND-0.21.2 breakage)

Live incident spanning framework-pt and shorty-s after the HTTPS/launcher
work and the LND 0.18.4→0.21.2 pin bump. Root causes found on real nodes;
status updated as work lands. Each fix ships with a regression test so the
same class cannot silently return.

## A. Root causes (all verified live)

| # | Symptom | Root cause |
|---|---------|-----------|
| A1 | LND sends fail "Payment failed: Not Found" | LND 0.21 **removed** the deprecated `/v1/channels/transactions` REST route; backend still called it. Receive was fine; the "Failed to fetch" on framework-pt was A3 masking it. |
| A2 | Shorty NPM restart-loops (counter 3176) | Manifest conversion (fc68c5b6) dropped (a) the `/etc/letsencrypt` mount NPM's s6 boot demands, and (b) `NET_BIND_SERVICE` — its internal nginx binds 80/443/81 and the orchestrator runs `--cap-drop=ALL`. |
| A3 | framework-pt: every `/rpc/v1` fetch CORS-blocked, "Failed to fetch", dashboard "not responding", mempool/indeehub frames broken | nginx sent `Strict-Transport-Security: max-age=31536000; includeSubDomains` on **HTTPS**; browsers cached it, then silently upgraded the still-open **http** dashboard's fetches/frames to https → scheme change = cross-origin → CORS block. HTTP is a supported mode on purpose (self-signed cert, /ca.crt flow). |
| A4 | Mempool/IndeeHub/bitcoin-UI frames stay `http://` on HTTPS pages (mixed content, "does not connect") | `portAuth()` looked the launch port up under the launch alias (`mempool-web`, `lnd`, `bitcoin-knots`…); the signed catalog declares those ports under the manifest id that owns them (`archy-mempool-web`, `lnd-ui`, `bitcoin-ui`) → miss → launcher fell back to http. Cache also only warmed in Store/Discover views. |
| A5 | IndeeHub nostr sign-in dead over HTTPS | NIP-07 bridge compared `event.origin` for strict equality with the stored (http) app URL and replied to the **stored** URL as postMessage targetOrigin — both break when the frame was scheme-upgraded. |
| A6 | Portainer "disappeared" after restart/update, then demands a setup token "see server logs" | Update to 2.45.0 recreated the container; on a fresh DB Portainer ≥2.21 mints a one-time setup token printed ONLY in container logs — hostile appliance UX. The "disappearance" was the recreate + this unknown-token first screen. |

## B. Fixes (code)

| Fix | Files | Status |
|-----|-------|--------|
| B1 LND pay via `Router.SendPaymentV2` (`/v2/router/send`), pending-status + actionable failure reasons preserved | `core/archipelago/src/api/rpc/lnd/payments.rs` (+ unit tests) | ✅ code |
| B2 Portainer setup token surfaced in the existing credentials interstitial (`package.credentials` → AppSidebar card with copy) | `core/archipelago/src/api/rpc/package/install.rs` (+ unit tests) | ✅ code |
| B3 HSTS: none on :80, `max-age=0` on :443 (actively clears cached policy) | `image-recipe/configs/nginx-archipelago.conf` | ✅ code |
| B4 NPM manifest: `/etc/letsencrypt` mount + `NET_BIND_SERVICE` | `apps/nginx-proxy-manager/manifest.yml` | ✅ code |
| B5 `portAuth` alias resolution + unanimous port-wide fallback | `neode-ui/src/views/discover/curatedApps.ts` | ✅ code |
| B6 Catalog cache warmed at dashboard bootstrap | `neode-ui/src/App.vue` | ✅ |
| B7 NIP-07 bridge: host/port equality + reply to `event.origin` | `neode-ui/src/stores/appLauncher.ts` ✅ · `neode-ui/src/views/appSession/useNostrBridge.ts` ✅ | ✅ |
| B8 Stale LND 0.18.4 refs in test expectations | `tests/lifecycle/remote-lifecycle.sh` | ✅ |

## C. Regression tests ("never again")

| Test | Guards | Status |
|------|-------|--------|
| C1 Rust: router v2 response shape, nested errors, failure reasons | B1 | ✅ |
| C2 Rust: setup-token log extraction (live-captured 2.45.0 line shape) | B2 | ✅ |
| C3 bats: `lnd-api-compat` — POST `/v2/router/send` on the running LND must answer (never 404) | B1 vs image skew at gate time | ✅ (route probe verified live on shorty: HTTP 500 ≠ 404) |
| C4 bats: nginx must NOT send HSTS on :80; :443 must send `max-age=0` | B3 | ✅ |
| C5 neode-ui unit: portAuth alias + unanimous-scan (incl. bitcoin-knots→8334 https) | B5/B4-mixed-content | ✅ (6 tests) |
| C6 neode-ui unit: bridge origin equality ignores scheme | B7 | ✅ (2 tests) |

Backend suites: 34 targeted Rust tests green (payments v2 shape, setup-token
extraction, lnd wallet/info regressions); middleware/dispatcher suite green;
full neode-ui suite green (62 tests in the touched areas); production bundle
built and verified to embed the alias fix. `cargo fmt` applied.

## D. Deploy & live verification

| Step | Status |
|------|--------|
| D1 shorty NPM crash-loop stopped cleanly (user-stopped marker; public hosts keep serving via host nginx mirror) | ✅ 12:52Z |
| D2 shorty live nginx HSTS patch + reload | ✅ verified: :80 and :443 both answer `max-age=0` |
| D3 Regenerate catalog (releases/app-catalog.json + store copies) | ✅ semantic diff = exactly the two NPM fixes |
| D4 **User runs `scripts/sign-catalog.sh`** (signer built at /tmp/archy-sign-bin) | ⬜ waiting on mnemonic |
| D5 Commit + push (origin + gitea-vps2 OTA mirror) | ✅ 6 commits pushed (signed catalog commits after D4) |
| D6 Release v1.8.9-alpha: `scripts/create-release.sh 1.8.9-alpha` (mnemonic) → `scripts/publish-release-assets.sh 1.8.9-alpha gitea-vps2` | ⬜ waiting on mnemonic |
| D7 OTA on shorty-s + framework-pt (Update button; framework-pt has no SSH from here) | ⬜ |
| D8 shorty: clear the NPM user-stopped marker + Start (or it starts via the fixed catalog) | ⬜ |
| D9 framework-pt: Start Mempool — its containers are confirmed stopped (port 4080 refuses; gate answers on 7778/8334/50002/18083 so those apps will embed over https immediately) | ⬜ |
| D10 Post-deploy live checks: LND send+receive; mempool/IndeeHub/bitcoin-UI frames over https; NPM healthy + admin :8081; portainer token card on fresh DB; zero CORS errors | ⬜ |

framework-pt immediate workaround until its OTA lands: open the dashboard by
IP (`http://192.168.x.x`) instead of `framework-pt.local`, and/or clear the
cached policy once via `chrome://net-internals/#hsts` → Delete domain security
policies → `framework-pt.local`.