102 lines
8.5 KiB
Markdown
102 lines
8.5 KiB
Markdown
# Incident + follow-up tracker — 2026-09-01 (post-HTTPS-work, post-LND-0.21.2 breakage)
|
||
|
||
Live incident spanning framework-pt and shorty-s after the HTTPS/launcher
|
||
work and the LND 0.18.4→0.21.2 pin bump. Root causes found on real nodes;
|
||
status updated as work lands. Each fix ships with a regression test so the
|
||
same class cannot silently return.
|
||
|
||
## A. Root causes (all verified live)
|
||
|
||
| # | Symptom | Root cause |
|
||
|---|---------|-----------|
|
||
| A1 | LND sends fail "Payment failed: Not Found" | LND 0.21 **removed** the deprecated `/v1/channels/transactions` REST route; backend still called it. Receive was fine; the "Failed to fetch" on framework-pt was A3 masking it. |
|
||
| A2 | Shorty NPM restart-loops (counter 3176) | Manifest conversion (fc68c5b6) dropped (a) the `/etc/letsencrypt` mount NPM's s6 boot demands, and (b) `NET_BIND_SERVICE` — its internal nginx binds 80/443/81 and the orchestrator runs `--cap-drop=ALL`. |
|
||
| A3 | framework-pt: every `/rpc/v1` fetch CORS-blocked, "Failed to fetch", dashboard "not responding", mempool/indeehub frames broken | nginx sent `Strict-Transport-Security: max-age=31536000; includeSubDomains` on **HTTPS**; browsers cached it, then silently upgraded the still-open **http** dashboard's fetches/frames to https → scheme change = cross-origin → CORS block. HTTP is a supported mode on purpose (self-signed cert, /ca.crt flow). |
|
||
| A4 | Mempool/IndeeHub/bitcoin-UI frames stay `http://` on HTTPS pages (mixed content, "does not connect") | `portAuth()` looked the launch port up under the launch alias (`mempool-web`, `lnd`, `bitcoin-knots`…); the signed catalog declares those ports under the manifest id that owns them (`archy-mempool-web`, `lnd-ui`, `bitcoin-ui`) → miss → launcher fell back to http. Cache also only warmed in Store/Discover views. |
|
||
| A5 | IndeeHub nostr sign-in dead over HTTPS | NIP-07 bridge compared `event.origin` for strict equality with the stored (http) app URL and replied to the **stored** URL as postMessage targetOrigin — both break when the frame was scheme-upgraded. |
|
||
| A6 | Portainer "disappeared" after restart/update, then demands a setup token "see server logs" | Update to 2.45.0 recreated the container; on a fresh DB Portainer ≥2.21 mints a one-time setup token printed ONLY in container logs — hostile appliance UX. The "disappearance" was the recreate + this unknown-token first screen. |
|
||
|
||
## B. Fixes (code)
|
||
|
||
| Fix | Files | Status |
|
||
|-----|-------|--------|
|
||
| B1 LND pay via `Router.SendPaymentV2` (`/v2/router/send`), pending-status + actionable failure reasons preserved | `core/archipelago/src/api/rpc/lnd/payments.rs` (+ unit tests) | ✅ code |
|
||
| B2 Portainer setup token surfaced in the existing credentials interstitial (`package.credentials` → AppSidebar card with copy) | `core/archipelago/src/api/rpc/package/install.rs` (+ unit tests) | ✅ code |
|
||
| B3 HSTS: none on :80, `max-age=0` on :443 (actively clears cached policy) | `image-recipe/configs/nginx-archipelago.conf` | ✅ code |
|
||
| B4 NPM manifest: `/etc/letsencrypt` mount + `NET_BIND_SERVICE` | `apps/nginx-proxy-manager/manifest.yml` | ✅ code |
|
||
| B5 `portAuth` alias resolution + unanimous port-wide fallback | `neode-ui/src/views/discover/curatedApps.ts` | ✅ code |
|
||
| B6 Catalog cache warmed at dashboard bootstrap | `neode-ui/src/App.vue` | ✅ |
|
||
| B7 NIP-07 bridge: host/port equality + reply to `event.origin` | `neode-ui/src/stores/appLauncher.ts` ✅ · `neode-ui/src/views/appSession/useNostrBridge.ts` ✅ | ✅ |
|
||
| B8 Stale LND 0.18.4 refs in test expectations | `tests/lifecycle/remote-lifecycle.sh` | ✅ |
|
||
|
||
## C. Regression tests ("never again")
|
||
|
||
| Test | Guards | Status |
|
||
|------|-------|--------|
|
||
| C1 Rust: router v2 response shape, nested errors, failure reasons | B1 | ✅ |
|
||
| C2 Rust: setup-token log extraction (live-captured 2.45.0 line shape) | B2 | ✅ |
|
||
| C3 bats: `lnd-api-compat` — POST `/v2/router/send` on the running LND must answer (never 404) | B1 vs image skew at gate time | ✅ (route probe verified live on shorty: HTTP 500 ≠ 404) |
|
||
| C4 bats: nginx must NOT send HSTS on :80; :443 must send `max-age=0` | B3 | ✅ |
|
||
| C5 neode-ui unit: portAuth alias + unanimous-scan (incl. bitcoin-knots→8334 https) | B5/B4-mixed-content | ✅ (6 tests) |
|
||
| C6 neode-ui unit: bridge origin equality ignores scheme | B7 | ✅ (2 tests) |
|
||
|
||
Backend suites: 34 targeted Rust tests green (payments v2 shape, setup-token
|
||
extraction, lnd wallet/info regressions); middleware/dispatcher suite green;
|
||
full neode-ui suite green (62 tests in the touched areas); production bundle
|
||
built and verified to embed the alias fix. `cargo fmt` applied.
|
||
|
||
## D. Deploy & live verification
|
||
|
||
| Step | Status |
|
||
|------|--------|
|
||
| D1 shorty NPM crash-loop stopped cleanly (user-stopped marker; public hosts keep serving via host nginx mirror) | ✅ 12:52Z |
|
||
| D2 shorty live nginx HSTS patch + reload | ✅ verified: :80 and :443 both answer `max-age=0` |
|
||
| D3 Regenerate catalog (releases/app-catalog.json + store copies) | ✅ semantic diff = exactly the two NPM fixes |
|
||
| D4 **User runs `scripts/sign-catalog.sh`** (signer built at /tmp/archy-sign-bin) | ✅ catalog signed + committed + pushed |
|
||
| D5 Commit + push (origin + gitea-vps2 OTA mirror) | ✅ 9 commits pushed |
|
||
| D6 Release v1.8.9-alpha: `scripts/create-release.sh 1.8.9-alpha` (mnemonic) → `scripts/publish-release-assets.sh 1.8.9-alpha gitea-vps2` | ✅ PUBLISHED (tag v1.8.9-alpha, releases/manifest.json live, backend+frontend assets verified by the script) |
|
||
| D7 OTA on shorty-s + framework-pt (Update button; shorty is on 1.8.8-alpha, daily check — hit Update now) | ⬜ user action |
|
||
| D8 shorty: clear the NPM user-stopped marker + Start (or it starts via the fixed catalog) | ✅ NPM LIVE-HEALED via the signed catalog: unit regenerated with both fixes, container up, admin UI HTTP 200 on :8081 (verified 15:42Z) |
|
||
| D9 framework-pt: Start Mempool — its containers are confirmed stopped (port 4080 refuses; gate answers on 7778/8334/50002/18083 so those apps will embed over https immediately) | ⬜ |
|
||
| D10 Post-deploy live checks: LND send+receive; mempool/IndeeHub/bitcoin-UI frames over https; NPM healthy + admin :8081 ✅; portainer token card on fresh DB; zero CORS errors | ⬜ after nodes update |
|
||
|
||
## E. Follow-ups discovered during the incident (ride the NEXT release, v1.8.10+)
|
||
|
||
- **Funding-modal honesty fix landed after the v1.8.9 tag** (1464b1b2): the
|
||
Lightning "no channel" modal now states the node's real state — pending
|
||
channel confirming / balance on the far side / payment couldn't route /
|
||
genuinely no channels. Ships in the next release; needs its own
|
||
create-release run (one more mnemonic paste). The CHANGELOG entry for
|
||
v1.8.10 should carry it. Note the stale-direction defect it fixes: the
|
||
payment-failure mapper never set the direction, so a SEND failure showed
|
||
the RECEIVE-branch copy ("Receiving needs inbound liquidity…") — the exact
|
||
modal users saw while their node had a healthy 583k-outbound channel.
|
||
- Nodes poll for OTA updates on `daily_check` — after publishing, tell the
|
||
user to hit Update rather than wait for the next check.
|
||
- `origin` remote had a stale pushurl with a dead token (pushes failed);
|
||
fixed to the canonical repo URL, stale `~/.git-credentials` entry with an
|
||
encoded port removed.
|
||
|
||
## F. Post-v1.8.9 verification on shorty-s (2026-09-01 evening)
|
||
|
||
- v1.8.9 applied; payment pipeline confirmed live: a 400,000 sat payment
|
||
SUCCEEDED through the v2 router route; the 404s are gone.
|
||
- App gate serves TLS on 4080/8334/18083/50002 (401 gate pages over https) —
|
||
https app frames now answer. Mempool over https requires a hard refresh
|
||
(PWA precaches the old bundle).
|
||
- **"No route to the recipient" on sends is real**: the invoices being tested
|
||
are from framework-pt, whose only channel (peer "Sandwich Farm",
|
||
0224c955…) is flagged `disabled` on BOTH policy sides in the routing graph
|
||
after today's node churn — the peer connection never re-established
|
||
(LND's reconnect backoff can stretch to hours). A disabled edge is
|
||
unroutable in both directions, so payments to/from framework-pt fail
|
||
regardless of shorty's 583k outbound. Fix: `lncli connect` the peer, wait
|
||
for the channel_update to re-enable the edge (~minutes), then re-test.
|
||
- The 577k attempt earlier failed for a different, correct reason: it exceeded
|
||
the channel's spendable balance (583,542 − 9,850 reserve ≈ 573k max).
|
||
|
||
framework-pt immediate workaround until its OTA lands: open the dashboard by
|
||
IP (`http://192.168.x.x`) instead of `framework-pt.local`, and/or clear the
|
||
cached policy once via `chrome://net-internals/#hsts` → Delete domain security
|
||
policies → `framework-pt.local`.
|