lfg2025/archy
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases 44 Wiki Activity

fix(nginx): self-heal fedimint asset rewrite on deployed nodes — HTTP + HTTPS (B13)

Browse Source 
The B13 template fix only fixed fresh ISOs. Already-deployed nodes keep their
old nginx config, where /app/fedimint/ proxies to :8175 without rewriting the
Guardian UI's root-rooted asset URLs (src="/assets/...", url("/assets/...")).
Those resolve against the SPA root: bg-network.jpg exists there by luck, but
app-icons/fedimint.jpg 404s (location /assets/ uses try_files =404) — the
visibly-broken icon.

bootstrap.rs::patch_nginx_conf now heals both paths on startup:
- Style A (main conf, HTTP): swaps the old single nostr-provider sub_filter tail
  for the full reroot set; byte-matches the shipped template.
- Style B (HTTPS app-proxy snippet): the snippet's fedimint block has no
  sub_filter and a per-node-varying trailing directive, so anchor on the unique
  :8175 proxy_pass and insert the reroot set after it (nginx ignores directive
  order). Snippet added to the bootstrap nginx loop (skipped on HTTP-only nodes).

missing_* flags are now gated on their splice anchors so the included snippet
neither attempts the main-conf-only patches nor logs warn-skips every boot.
Idempotent via the 'href="/' 'href="/app/fedimint/' marker.

Verified on .198 (both paths): fedimint app-icon 404 -> 200 image/jpeg; nginx -t
OK; containers survived restart (Quadlet); idempotent steady state, no warn spam.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago 2026-06-15 18:03:04 -04:00
parent a50b6df21b
commit 987a961f4a
2 changed files with 81 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

69
core/archipelago/src/bootstrap.rs
View File

@ -32,6 +32,12 @@ const DOCTOR_TIMER_PATH: &str = "/etc/systemd/system/archipelago-doctor.timer";
const NGINX_CONF_PATH: &str = "/etc/nginx/sites-available/archipelago"; const NGINX_CONF_PATH: &str = "/etc/nginx/sites-available/archipelago";
const NGINX_ENABLED_CONF_PATH: &str = "/etc/nginx/sites-enabled/archipelago"; const NGINX_ENABLED_CONF_PATH: &str = "/etc/nginx/sites-enabled/archipelago";
/// Per-app proxy snippet included by the HTTPS (:443) server block. Carries its
/// own `/app/fedimint/` location, so it needs the same B13 asset-rewrite heal as
/// the main conf — browsers reach fedimint over HTTPS via this snippet. Absent on
/// HTTP-only nodes, in which case the bootstrap loop skips it.
const NGINX_HTTPS_SNIPPET_PATH: &str =
"/etc/nginx/snippets/archipelago-https-app-proxies.conf";
const RUNTIME_ASSETS_DIR: &str = "/opt/archipelago/web-ui/archipelago-runtime"; const RUNTIME_ASSETS_DIR: &str = "/opt/archipelago/web-ui/archipelago-runtime";
/// Inserted into every server block of the nginx config that lacks the /// Inserted into every server block of the nginx config that lacks the
@ -56,6 +62,26 @@ const NGINX_LND_PROXY_BLOCK: &str = "\n # LND REST proxy — backend handles
/// block in image-recipe/configs/nginx-archipelago.conf. /// block in image-recipe/configs/nginx-archipelago.conf.
const NGINX_PEER_CONTENT_BLOCK: &str = "\n # Peer content streaming proxy (B3) — Range-streams a peer's media file\n location /api/peer-content/ {\n proxy_pass http://127.0.0.1:5678;\n proxy_http_version 1.1;\n proxy_set_header Host $host;\n proxy_set_header Cookie $http_cookie;\n proxy_set_header Range $http_range;\n proxy_buffering off;\n proxy_connect_timeout 10s;\n proxy_read_timeout 120s;\n error_page 502 503 = @backend_unavailable;\n error_page 504 = @backend_timeout;\n }\n"; const NGINX_PEER_CONTENT_BLOCK: &str = "\n # Peer content streaming proxy (B3) — Range-streams a peer's media file\n location /api/peer-content/ {\n proxy_pass http://127.0.0.1:5678;\n proxy_http_version 1.1;\n proxy_set_header Host $host;\n proxy_set_header Cookie $http_cookie;\n proxy_set_header Range $http_range;\n proxy_buffering off;\n proxy_connect_timeout 10s;\n proxy_read_timeout 120s;\n error_page 502 503 = @backend_unavailable;\n error_page 504 = @backend_timeout;\n }\n";
/// B13 — Fedimint UI asset rewrite. Pre-fix nodes proxy /app/fedimint/ with only
/// the nostr-provider injection (`sub_filter_once on`), so the UI's root-rooted
/// CSS/JS asset URLs (href="/…", url("/…")) miss the proxy and load the SPA shell
/// → unstyled UI. We swap that single sub_filter for the full rewrite set that
/// reroots every asset URL under /app/fedimint/. NEW matches the canonical block
/// in image-recipe/configs/nginx-archipelago.conf byte-for-byte so self-healed
/// nodes converge to the same config fresh ISOs ship with.
const NGINX_FEDIMINT_OLD: &str = " sub_filter_once on;\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';\n }\n location /app/fedimint-gateway/ {";
const NGINX_FEDIMINT_NEW: &str = " sub_filter_types text/css application/javascript application/json;\n sub_filter_once off;\n sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n sub_filter \"href='/\" \"href='/app/fedimint/\";\n sub_filter \"src='/\" \"src='/app/fedimint/\";\n sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n sub_filter \"url('/\" \"url('/app/fedimint/\";\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';\n }\n location /app/fedimint-gateway/ {";
/// B13 Style B — the HTTPS app-proxy snippet's fedimint block has NO sub_filter
/// at all (older than the main conf's), and the directive that follows it varies
/// per node (fedimint-gateway vs tailscale), so a full-block match is unreliable.
/// Instead we anchor on the unique :8175 proxy_pass (fedimint is the only block
/// proxying there) and insert the reroot set right after it — directive order
/// inside a location block is irrelevant to nginx. Idempotent via the same
/// `href="/app/fedimint/` marker the main-conf heal leaves behind.
const NGINX_FEDIMINT_SNIPPET_ANCHOR: &str = "proxy_pass http://127.0.0.1:8175/;";
const NGINX_FEDIMINT_SNIPPET_INSERT: &str = "proxy_pass http://127.0.0.1:8175/;\n proxy_set_header Accept-Encoding \"\";\n sub_filter_types text/css application/javascript application/json;\n sub_filter_once off;\n sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n sub_filter \"href='/\" \"href='/app/fedimint/\";\n sub_filter \"src='/\" \"src='/app/fedimint/\";\n sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n sub_filter \"url('/\" \"url('/app/fedimint/\";\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';";
/// Entry point called from main startup. Never returns an error to the caller — /// Entry point called from main startup. Never returns an error to the caller —
/// failing to bootstrap host artifacts must not prevent the backend from serving. /// failing to bootstrap host artifacts must not prevent the backend from serving.
pub async fn ensure_doctor_installed() { pub async fn ensure_doctor_installed() {
@ -511,7 +537,11 @@ async fn run_nginx() -> Result<bool> {
let mut changed = false; let mut changed = false;
let mut patched_paths = Vec::<PathBuf>::new(); let mut patched_paths = Vec::<PathBuf>::new();
for path in [NGINX_CONF_PATH, NGINX_ENABLED_CONF_PATH] { for path in [
NGINX_CONF_PATH,
NGINX_ENABLED_CONF_PATH,
NGINX_HTTPS_SNIPPET_PATH,
] {
let candidate = Path::new(path); let candidate = Path::new(path);
if !candidate.exists() { if !candidate.exists() {
debug!("{} missing — skipping nginx bootstrap", path); debug!("{} missing — skipping nginx bootstrap", path);
@ -541,16 +571,29 @@ async fn patch_nginx_conf(path: &str) -> Result<bool> {
let content = fs::read_to_string(path) let content = fs::read_to_string(path)
.await .await
.with_context(|| format!("read {}", path))?; .with_context(|| format!("read {}", path))?;
let missing_app_catalog = !content.contains("location /api/app-catalog"); // Each "missing" flag is gated on the splice anchor actually being present,
let missing_bitcoin_status = !content.contains("location /bitcoin-status"); // so an included snippet that legitimately has none of these endpoints (the
let missing_lnd_proxy = !content.contains("location /proxy/lnd/"); // HTTPS app-proxy snippet) neither tries to patch them nor logs warn-skips on
let missing_peer_content = !content.contains("location /api/peer-content"); // every boot — it falls through to the fedimint heal alone.
let has_lnd_anchor = content.contains(" location /lnd-connect-info {")
|| content.contains(" location /electrs-status {");
let missing_app_catalog = content
.contains(" # DWN endpoints — peer access over Tor (no auth)")
&& !content.contains("location /api/app-catalog");
let missing_bitcoin_status = content.contains(" location /electrs-status {")
&& !content.contains("location /bitcoin-status");
let missing_lnd_proxy = has_lnd_anchor && !content.contains("location /proxy/lnd/");
let missing_peer_content = has_lnd_anchor && !content.contains("location /api/peer-content");
let has_lnd_dup_cors = content.contains(NGINX_LND_DUP_CORS); let has_lnd_dup_cors = content.contains(NGINX_LND_DUP_CORS);
// B13: fedimint block present but lacking the asset-rewrite sub_filters.
let needs_fedimint_css = content.contains("location /app/fedimint/")
&& !content.contains("'href=\"/' 'href=\"/app/fedimint/'");
if !missing_app_catalog if !missing_app_catalog
&& !missing_bitcoin_status && !missing_bitcoin_status
&& !missing_lnd_proxy && !missing_lnd_proxy
&& !missing_peer_content && !missing_peer_content
&& !has_lnd_dup_cors && !has_lnd_dup_cors
&& !needs_fedimint_css
{ {
return Ok(false); return Ok(false);
} }
@ -563,6 +606,22 @@ async fn patch_nginx_conf(path: &str) -> Result<bool> {
patched = patched.replace(NGINX_LND_DUP_CORS, ""); patched = patched.replace(NGINX_LND_DUP_CORS, "");
} }
if needs_fedimint_css {
// Style A (main conf): the block already injects nostr-provider, so swap
// its single-sub_filter tail for the full asset-rewrite set. No-op if the
// node's fedimint block doesn't match OLD.
patched = patched.replace(NGINX_FEDIMINT_OLD, NGINX_FEDIMINT_NEW);
// Style B (HTTPS app-proxy snippet): the block has no sub_filter to swap,
// so insert the reroot set after the unique :8175 proxy_pass. Guarded on
// the marker so it can never double-apply after Style A already healed.
if !patched.contains("'href=\"/' 'href=\"/app/fedimint/'") {
patched = patched.replace(
NGINX_FEDIMINT_SNIPPET_ANCHOR,
NGINX_FEDIMINT_SNIPPET_INSERT,
);
}
}
if missing_lnd_proxy { if missing_lnd_proxy {
// Prefer the `/lnd-connect-info` anchor (present since 2026-03-17); fall // Prefer the `/lnd-connect-info` anchor (present since 2026-03-17); fall
// back to `/electrs-status` (since 2026-03-08) for even older configs. // back to `/electrs-status` (since 2026-03-08) for even older configs.

32
tests/production-quality/TRACKER.md
View File

@ -8,21 +8,13 @@ cd ~/Projects/archy && git fetch gitea-vps2 && git checkout main && git reset --
``` ```
Then continue from "IN PROGRESS" below. Then continue from "IN PROGRESS" below.
**Committed & ready for .97 (vps2 main):** B5 (LND CORS, verified .116/.198/.103), B1, B2, B4, B14, B21, B3 (incl. /api/peer-content nginx via bootstrap), B15, B7. B6 pruned-gate already live. **Committed & ready for .97 (vps2 main):** B5 (LND CORS, verified .116/.198/.103), B1, B2, B4, B14, B21, B3 (incl. /api/peer-content nginx via bootstrap), B15, B7, **B13 (fedimint CSS self-heal — main conf + HTTPS snippet, verified .198 both paths app-icon 404→200)**. B6 pruned-gate already live.
**IN PROGRESS — B13 Fedimint CSS:** nginx template + https snippet committed (fixes FRESH ISOs). REMAINING (resume here): add the bootstrap self-heal for already-deployed nodes, then BUILD (cargo build --release -p archipelago — ALWAYS check EXIT 0 before commit), sideload to .198, verify `curl http://192.168.1.198/app/fedimint/` styled + an asset returns CSS not HTML. Exact code to add in core/archipelago/src/bootstrap.rs: **IN PROGRESS — pick up at B12.** B13 DONE (committed this session; bootstrap.rs self-heals both the main conf and the HTTPS app-proxy snippet — see B13 entry below for full verification). REMAINING:
1. **B12** (mempool host detect — stacks.rs:1278 hardcodes CORE_RPC_HOST=bitcoin-knots; fails on bitcoin-core nodes → dynamic host detect; backend, medium risk, test .116).
2. Then **B16** (bitcoin status retain — UI-test), **B6** no-node-present half, **B14b** (FIPS reachability depth), **B22/B23** (peer download + group chat — need live repro), B9/B10/B11/B17/B18/B19, B8 (low), B20 (mesh-headers feature).
```rust Note: .198 is currently running a sideloaded .97-dev binary (md5 4c83803d, built from this B13 commit) — NOT an official release. Reflashing/OTA will replace it.
// after NGINX_PEER_CONTENT_BLOCK const:
const NGINX_FEDIMINT_OLD: &str = " sub_filter_once on;\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';\n }\n location /app/fedimint-gateway/ {";
const NGINX_FEDIMINT_NEW: &str = " sub_filter_types text/css application/javascript application/json;\n sub_filter_once off;\n sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n sub_filter \"href='/\" \"href='/app/fedimint/\";\n sub_filter \"src='/\" \"src='/app/fedimint/\";\n sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n sub_filter \"url('/\" \"url('/app/fedimint/\";\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';\n }\n location /app/fedimint-gateway/ {";
// in patch_nginx_conf(): add to guard + body:
// let needs_fedimint_css = content.contains("location /app/fedimint/") && !content.contains("'href=\"/' 'href=\"/app/fedimint/'");
// (add `&& !needs_fedimint_css` to the early-return guard)
// if needs_fedimint_css { patched = patched.replace(NGINX_FEDIMINT_OLD, NGINX_FEDIMINT_NEW); }
```
**Then, in priority order:** B12 (mempool host detect — stacks.rs:1278), B16 (bitcoin status retain — UI-test), B6 no-node-present half, B14b (FIPS reachability depth), B22/B23 (peer download + group chat — need live repro), B9/B10/B11/B17/B18/B19, B8 (low), B20 (mesh-headers feature).
**Ship .97 when ready:** ./scripts/create-release.sh 1.7.97-alpha (curate CHANGELOG ≥3 layman bullets first + run scripts/sync-whats-new.py; SKIP_RELEASE_TESTS=1 only for the 2 known-flaky vitest timing tests) → scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2 → git push gitea-vps2 main + tag. (gitea-local push fails: token rejected — non-blocking.) **Ship .97 when ready:** ./scripts/create-release.sh 1.7.97-alpha (curate CHANGELOG ≥3 layman bullets first + run scripts/sync-whats-new.py; SKIP_RELEASE_TESTS=1 only for the 2 known-flaky vitest timing tests) → scripts/publish-release-assets.sh 1.7.97-alpha gitea-vps2 → git push gitea-vps2 main + tag. (gitea-local push fails: token rejected — non-blocking.)
@ -114,8 +106,18 @@ Apps meant to open in a new/external browser don't launch from the companion app
### B12 — Mempool not connecting — ROOT-CAUSED (stacks.rs:1278 hardcodes CORE_RPC_HOST=bitcoin-knots; fails on bitcoin-core nodes. Fix=dynamic host detect. Backend, medium risk, test .116) ### B12 — Mempool not connecting — ROOT-CAUSED (stacks.rs:1278 hardcodes CORE_RPC_HOST=bitcoin-knots; fails on bitcoin-core nodes. Fix=dynamic host detect. Backend, medium risk, test .116)
mempool can't reach the Bitcoin backend on some nodes. Investigate on .116. Check mempool→electrs→bitcoind wiring + deps. mempool can't reach the Bitcoin backend on some nodes. Investigate on .116. Check mempool→electrs→bitcoind wiring + deps.
### B13 — Fedimint UI not applying CSS — TODO ### B13 — Fedimint UI not applying CSS — FIXED + VERIFIED on .198 (both HTTP + HTTPS)
Actual Fedimint UI (not pre-sync) renders unstyled. Likely asset path / proxy base-href (assets rooted at `/` vs `/app/fedimint/`). PROGRESS: nginx /app/fedimint/ proxies to :8175 with sub_filter for nostr-provider but NO base-href/asset rewrite. Fedimint UI head has no static CSS link in first 1.2KB (likely JS-module-loaded assets rooted at /). NEXT: dump full fedimint HTML + check if assets request /assets/* (which hit the SPA, not :8175). Fix = inject <base href="/app/fedimint/"> via sub_filter OR rebuild app with correct base. Needs deeper per-app look (fresh context ok). Root cause confirmed: the Fedimint Guardian page (served by :8175) is a server-rendered status page with ~7.8KB INLINE CSS plus image assets referenced root-rooted (`src="/assets/img/app-icons/fedimint.jpg"`, `url("/assets/img/bg-network.jpg")`). Without an asset rewrite those `/assets/...` URLs resolve against the archipelago SPA root: `bg-network.jpg` happens to exist there (shared design asset → loaded by luck) but `app-icons/fedimint.jpg` does NOT → **404** (the broken/visibly-missing icon). The `location /assets/` block uses `try_files $uri =404`, so missing fedimint assets 404 rather than fall through.
Fix = nginx sub_filter set that reroots every root-rooted asset URL (`href="/`, `src="/`, `url("/`, and single-quote variants) under `/app/fedimint/`, plus `proxy_set_header Accept-Encoding ""` so the upstream doesn't gzip (sub_filter can't rewrite gzipped bodies). Shipped two ways:
- **Fresh ISOs** (committed a50b6df2): templates `image-recipe/configs/nginx-archipelago.conf` (HTTP) + `image-recipe/configs/snippets/archipelago-https-app-proxies.conf` (HTTPS).
- **Already-deployed nodes** (bootstrap self-heal, this commit): `core/archipelago/src/bootstrap.rs::patch_nginx_conf` now heals BOTH the main conf (Style A — swaps the old single nostr-provider sub_filter tail for the full reroot set, byte-matches the shipped template) AND the HTTPS app-proxy snippet (Style B — anchors on the unique `:8175` proxy_pass and inserts the reroot set; robust to the snippet's varying trailing directive). `missing_*` flags now gated on their splice anchors so the healed snippet early-returns cleanly (no per-boot warn-skips). Idempotent via the `'href="/' 'href="/app/fedimint/'` marker.
VERIFIED on .198 (sideloaded built binary, restart, async self-heal converged ~15s):
- HTTP `/app/fedimint/`: live conf healed byte-identical to template; app-icon **404→200 image/jpeg (41944b)**.
- HTTPS `/app/fedimint/` (snippet): healed; same app-icon **404→200**; bg-network 200; root `/assets/img/app-icons/fedimint.jpg` returns 200 **text/html** (SPA shell) — proving the reroot is necessary.
- `nginx -t` OK both times; containers survived restart (Quadlet); both files carry the marker exactly once (idempotent steady state); no warn spam in logs.
NOTE: self-healed snippet is functionally correct but NOT byte-identical to the fresh-ISO snippet template (insert-after-proxy_pass vs full block) — acceptable; nginx ignores directive order/whitespace.
### B15 — Bitcoin UI sync progress lags — FIXED (Home.vue poll 30s→10s). UI-confirm. ### B15 — Bitcoin UI sync progress lags — FIXED (Home.vue poll 30s→10s). UI-confirm.
Bitcoin UI doesn't update its sync progress fast enough even though the console clearly already has the block-height data. Likely a polling-interval / reactive-update gap between the status source and the UI. Bitcoin UI doesn't update its sync progress fast enough even though the console clearly already has the block-height data. Likely a polling-interval / reactive-update gap between the status source and the UI.