Adds a verification checklist (service running, nodogsplash bound to
br-tollgate not br-lan via the rendered config not just UCI, LAN/SSH
untouched, mint probes succeeding) plus notes on the dev-build test-mint
injection and the default-route race between a router's LAN interface
and the node's other uplinks before the router's own WAN/WISP is live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
Archipelago's Connect form only authenticates with an existing password;
it has no flow for setting one on a fresh, passwordless router. On the
node's kiosk display there's no visible tab bar, so Ctrl+T to open a new
tab to LuCI is the way to set it before Connect will work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
The install code was hardcoded to the Oct 2025 v0.2.0 release —
nine releases behind. Its changelog covers exactly the failures hit
live against archy-x250-pa3: a mint with an empty/broken keyset
crash-looped tollgate-wrt forever (v0.5.0 adds "graceful degradation
when Cashu mints fail"), and the bundled captive-portal JS had zero
CBOR support, hard-rejecting the cashuB (NUT-00 V4) tokens modern
wallets like Minibits generate by default.
Also: v0.5.0 publishes native .apk packages for aarch64_cortex-a53
and x86_64. install_tollgate_apk_native now prefers those directly
(apk add handles deps/postinst/uci-defaults itself) instead of always
falling back to the manual ar/tar .ipk extraction dance, which only
exists because earlier releases had no native apk build at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
Observed live on archy-x250-pa3: right after WAN reconnects (fresh
boot or WAN reconfigure), the first Install attempt can fail with
"apk update failed ... router may have no internet access" purely
because the WiFi-uplink STA association hasn't finished yet — it's
not a real error, just retry a few seconds later. Also cross-referenced
the now-fixed /usr/bin/opkg hardcoding bug for anyone hitting it on an
older build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
provisionTollgate/saveTollgateConfig/scanWifi/configureWan all fell
back to the Connect form's local refs (host/sshUser/sshPassword) when
connectedParams was null. Those refs only get populated if the form
was actually submitted this session — on a normal page load the
router reconnects via the server-persisted config instead, leaving
sshPassword at its default ''. Sending that as an explicit
(empty-but-present) ssh_password overrides the backend's saved-config
fallback, so every action auths with a blank password instead of the
real saved one.
Added authParams(): omit host/ssh_user/ssh_password entirely unless
connectedParams is actually set, same as the status poll already does.
Caught live: dropbear on archy-x250-pa3's router logged a single bad
password attempt at the exact moment "Install TollGate" was clicked,
sandwiched between periodic status-poll connections succeeding with
the real saved password.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
opkg_check() and every opkg/apk invocation hardcoded /usr/bin/opkg and
/usr/bin/apk. Official OpenWrt images don't all symlink /bin into
/usr/bin — the glinet_gl-mt3000 24.10.2 build keeps them as separate
real directories with opkg living in /bin — so the check silently
missed a perfectly normal install and TollGate provisioning failed
with "this router's firmware may not support package management".
Switched every call to resolve through the router's own $PATH
(command -v / bare opkg / apk) instead. Reproduced and fixed live
against archy-x250-pa3, 2026-09-05.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
Step 4 described a single "Provision TollGate" action that prompts for
price/step/mint upfront. The real UI (OpenWrtGateway.vue) doesn't work
that way: "Install TollGate" is a one-click action with no config form
that installs with defaults, and price/step/mint/enabled are only
editable afterward via a separate "Edit" panel. Caught while walking
through a live install.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
Worked example for the Beryl AX (GL-MT3000, mediatek/filogic) verified
against the OpenWrt wiki and firmware selector: exact sysupgrade image
filename, GL.iNet UI / LuCI flash path, post-flash SSH state, and the
U-Boot recovery procedure if the flash goes sideways.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
Walks a node operator through pairing an OpenWrt router over SSH,
running the WAN/WISP wizard, and provisioning TollGate pay-as-you-go
WiFi — plus an RPC/architecture reference for developers. Distills
the openwrt crate, RPC handlers, and Vue panel into user-facing steps
that didn't exist anywhere in docs/ before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0176RpCxFNS9ZaSJjL72W9Z5
The restricted-RPC port (18090) was `auth: none`, which the app gate
treats as fully exempt — no onion, no takeover, LAN/Tailscale IP only.
Flip it to `auth: open`: the gate still binds the external addresses
and fronts a Tor onion for the port, just without a dashboard login
challenge, since Monero wallet clients (Feather, monero-wallet-rpc,
GUI) speak plain HTTP JSON-RPC and can't hold a session cookie.
P2P (18183) stays `none` — no reason to Tor-front raw gossip.
Regenerated releases/app-catalog.json (unsigned) to embed the updated
manifest; needs scripts/sign-catalog.sh before it takes effect on any
node, since origin (catalog) wins over disk for catalog-covered apps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NZnsiMtyJxiJBuvv7yLPUF
The default manifest baked in the exact broken config found on an
affected fleet node: no fast_sync (defaults false, forcing full ring-sig/
RandomX verification on every block) and target_max_memory capped at
~2.8GiB, which starved cuprated's DB cache into constant eviction/flush
(595GB/24h of block I/O on a node just appending ~2MB blocks every 2
minutes). A reference node with fast_sync = true and an 8GiB cache ran
at 2.8% CPU at the same chain height and block rate.
Set fast_sync = true and target_max_memory = 8GiB to match the healthy
reference config, and raise resources.memory_limit from 4Gi to 10Gi so
the container still has headroom above the larger cache.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RR7jRaicvqsJaqQQ92jpPQ
v1.8.9's move to Router.SendPaymentV2 shipped without fee_limit_sat,
and the v2 route treats an ABSENT fee limit as zero allowed fees.
Every real route carries a routing fee (the 2-hop route here: 1.5
sats), so the pathfinder rejected them all and the wallet answered
"No route to the recipient" on EVERY send — all day, on healthy
channels with plenty of liquidity both ways.
The router debug log makes it unambiguous:
wallet payment (v1.8.9 backend): fee_limit=0 mSAT -> no route
same payment by hand (lncli --fee_limit=100): fee_limit=100000 mSAT -> settles in 0.65s
My earlier "pipeline verified" claim was wrong — the manual lncli
verification set a fee limit by hand and masked this exact bug. The
400k that succeeded this morning went through the pre-update backend
on the pre-update LND.
Payments now carry lncli's own default budget — the payment amount
(100%), preferring the payer-supplied amount for zero-value invoices
and the invoice's own amount otherwise, with a nominal floor so the
limit can never be zero. Unit-pinned so it cannot regress.
LND normally reconnects channel peers after a restart, but not reliably:
after long or repeated downtime (an app update, a node reboot,
reconciler churn) the peer link can stay down for hours while BOTH
endpoints keep the channel flagged disabled in the routing graph. The
node looks perfectly healthy, the wallet shows balance, and every
payment in either direction fails "no route to the recipient" —
observed live on framework-pt (2026-09-01): its only channel sat
disabled on both policy sides for ~17 hours after the LND 0.21.2
update, while shorty had 583k spendable and the user was told, by a
mis-mapped modal, that they had 'no payment channel'.
The channel graph is desired state — every open channel should have a
live peer connection. A daemon-side watchdog now enforces it:
- every 2 minutes, list channels + peers over LND REST
- for each channel whose remote peer is not connected, look the peer's
advertised addresses up in the public graph and dial one
- per-peer retries throttled to 10 minutes so an unreachable peer is
not hammered; 'already connected' counts as done; a peer with no
advertised address is logged once per pass (cannot be dialed)
- no-ops quietly on nodes without LND (missing macaroon) and while a
wallet is locked (503 body has no channels)
Unit tests pin the selection against the live REST shapes
(remote_pubkey in /v1/channels vs pub_key in /v1/peers).
v1.8.10 CHANGELOG + What's New entries staged so the next release run
is clean first time.
"LND thinks I do not have a channel" while the wallet showed plenty of
liquidity (framework-pt, 2026-09-01): the send gate sums outbound over
FULLY-OPEN channels only, which is correct — a just-opened channel
sits in LND's pending list until it has ~3 confirmations, and an
open channel can have all its balance on the far side — but the modal
then claimed the node had NO channel at all, in every one of those
states, and pointed the user at opening another one.
The gate already fetched the full channel list; it now records WHY
liquidity is zero and the modal says the truth per state:
- pending channels -> "your new channel is waiting for on-chain
confirmations, it unlocks automatically, nothing is needed from you"
(and no "Open a channel" button — that would send the user to fix
a problem they don't have, possibly opening a second channel)
- open channels, zero on the needed side -> "balance is on the far
side — you can receive but there's nothing to send right now"
- payment refused with a routing/liquidity error -> says so, instead
of claiming no channels
- only a genuinely channel-less node keeps the open-one guidance
Eleven unit tests pin the state machine, including the regression
case (pending-only -> 'pending', not 'none') and fail-open on RPC
errors.
The release gate requires the freshly-built bundle to embed the new
version, and the version reaches the bundle through the What's New
modal in AccountInfoSection — there was no v1.8.9-alpha block yet, so
create-release.sh correctly refused to ship a bundle that looked stale.
This adds the block (the user-facing summary of today's LND/HTTPS/
launcher/NPM/Portainer fixes) and carries the version bumps the
aborted run had already written (Cargo.toml, Cargo.lock, package.json,
package-lock), so the re-run starts from a clean tree.
Verified: npm run build now produces assets containing 1.8.9-alpha
(Settings chunk), i.e. the exact check the script runs passes.
Regenerated from the fixed apps/nginx-proxy-manager/manifest.yml (the
only semantic change vs the previous signed catalog) and signed with
the release-root key. Catalog-covered nodes pick this up on their next
hourly fetch and the NPM start/die loop ends: s6 gets its /etc/letsencrypt
mount back and the internal nginx can bind 80/443/81 again under
--cap-drop=ALL.
Root-cause table, fix inventory, regression-test inventory and the
deploy/live-verification checklist for today's fleet incident — written
as the working record while the fixes land, so the deploy + verify steps
can be checked off against real nodes rather than memory. CHANGELOG
carries the user-facing notes for the release these fixes ship in.
Three launcher/bridge defects combined to make HTTPS dashboards look
broken while HTTP ones worked:
1. portAuth() looked the launch port up under the name the user clicks
('mempool-web', 'lnd', 'bitcoin-knots'…), but the signed catalog
declares those ports under the manifest id that owns them
(archy-mempool-web, lnd-ui, bitcoin-ui). The lookup missed,
portIsGateFronted answered false, and an HTTPS dashboard handed app
frames http:// URLs — blocked as mixed content: mempool and IndeeHub
'did not connect', bitcoin knots/core opened http:// in a new tab.
Resolution now follows launch aliases, then a port-wide catalog scan
that only answers when every declarer of that port agrees (a port
any app publishes as plain HTTP is never upgraded to https).
2. The signed-catalog cache was only warmed by the Store/Discover
views, so a user who went straight to My Apps launched apps with an
empty cache. Warmed at dashboard mount now — fetchAppCatalog()
already memoizes with a 1h TTL.
3. The NIP-07 bridge compared event.origin for strict equality with the
recorded (http) app URL and replied to the recorded URL as the
postMessage targetOrigin — both break the moment a frame is scheme-
upgraded (cached HSTS did exactly that): every nostr request was
silently dropped and replies to the stale origin threw. The bridge
now matches host+port (scheme deliberately ignored) and always
replies to event.origin — the frame's real origin.
Unit tests cover alias resolution (incl. bitcoin-knots→8334→https),
the conservative port-scan, and scheme-agnostic sender matching.
Converting Nginx Proxy Manager to a platform manifest (fc68c5b6) dropped
two things its image hard-requires, and the result was an endless
start/die loop — shorty-s watched it restart 3,176 times:
1. /etc/letsencrypt mount: NPM's s6 'prepare' service refuses to boot
without it ('ERROR: /etc/letsencrypt is not mounted!'). Mounted from
the same persistent app directory as before
(/var/lib/archipelago/nginx-proxy-manager/letsencrypt), so existing
certificates are preserved — no data moves, no migration.
2. NET_BIND_SERVICE: NPM's internal nginx listens on 80, 443 AND 81,
and the orchestrator runs --cap-drop=ALL. The legacy podman-run path
defaulted to the full capability set (and the legacy repair path in
package/config.rs always listed it), which is why this only broke
once the manifest became the source of truth.
The signed catalog embeds manifests with origin-wins semantics, so the
catalog carries the fix for every catalog-covered node — regenerate it
here (plus the generated store/launcher-port artifacts, which also pick
up drift from bf6ef964's retired apps). Catalog re-signing follows the
usual ceremony.
The HTTPS server block sent Strict-Transport-Security:
max-age=31536000; includeSubDomains. Browsers that visited HTTPS once
cached the policy and then silently upgraded the still-open HTTP
dashboard's fetches and frames to https — a scheme change is
cross-origin, so every /rpc/v1 call died 'No Access-Control-Allow-
Origin header' while the node was perfectly healthy (framework-pt
2026-09-01: the 'Failed to fetch' storm, dashboard 'not responding',
every app frame mixed-content-blocked).
Plain HTTP is a supported access mode BY DESIGN on this platform: the
node's certificate is optional and self-signed (Settings → Node
certificate, /ca.crt flow), and setup-node-ca.sh deliberately keeps
port 80 serving for devices that haven't installed the CA. So:
- port 80 sends no HSTS at all (with the rationale inline)
- port 443 sends max-age=0, which ACTIVELY DELETES the policy already
cached by affected browsers — leaving it absent would have kept every
stranded browser broken for a year
tests/lifecycle/bats/nginx-hsts.bats pins all three properties at the
gate: no live policy on :80, max-age=0 (never 31536000) on :443, and
no long-lived pin anywhere in the deployed config.
Portainer >=2.21 no longer lets whoever loads the page first claim the
admin account: on a fresh install it mints a one-time setup token and
prints it ONLY to the server logs. On an appliance that is a dead end —
'check the Portainer server logs' is exactly what a user cannot follow,
and after the 2.45.0 update it made a freshly restarted Portainer look
broken ('disappeared', then demands a token nobody can find).
package.credentials — the same RPC that powers the login-credentials
card on the app page — now extracts the setup_token line from
portainer's recent container logs and hands it over with the existing
copy-button treatment, titled and explained for a first-time user. The
token stops being printed once setup completes, and any container
recreate drops the log line, so the card disappears on its own and no
dead token lingers. Parsing is a pure, unit-tested scan against the
live-captured 2.45.0 log shape (64 hex chars after setup_token=).
LND 0.21.2 removed the deprecated Lightning.SendPaymentSync REST route
(/v1/channels/transactions). The backend still called it, so every
Lightning send answered literal HTTP 404 and the wallet UI reported
'Payment failed: Not Found' fleet-wide right after the pin bump —
receive worked, which made it look intermittent.
Pay through the supported Router.SendPaymentV2 route (/v2/router/send)
instead, keeping the existing contract with the UI:
- single-record responses (no_inflight_updates) unwrapped from the
grpc-gateway result envelope, transport errors from the nested error
- a slow multi-hop payment still resolves as pending + payment hash
(only LND may declare failure), never a false 'Payment failed'
- LND's failure_reason codes translated to the same plain-language
advice, invoice-expiry still says 'ask for a fresh invoice'
Guard it at the gate: tests/lifecycle/bats/lnd-api-compat.bats POSTs a
deliberately-invalid invoice to /v2/router/send on the RUNNING LND and
fails if the route answers 404 — the image/backend skew that shipped
silently last time because no test ever spoke the payment endpoint.
Also bumps the stale lnd image expectation in remote-lifecycle.sh.
The previous commit on this branch copied cuprated's raw
--generate-config defaults (stdout=info, file=debug, max_log_files=7)
verbatim. Turns out that's the wrong reference: compared against
ssmithx@archy-dev-pa:/home/ssmithx/cuprate/Cuprated.toml — the actual
dev config this app was built and tested against — file logging is
meant to run at "info" with 14 rotated files, not the binary's raw
"debug"/7. Confirmed live on amishparadise: podman logs (stdout) was
already clean at info, but the on-disk file log
(.local/share/cuprate/logs/<date>) was flooding with per-peer DEBUG
gossip (~400KB in 2 minutes) because it inherited the binary default
instead of the intended one.
Left the resource-tuning knobs in the reference config (8GB
target_max_memory, tokio/rayon thread counts, P2P connection counts,
explicit reader_threads) out of this file — those were sized for
ssmithx's dev box and don't apply here; this manifest's
target_max_memory already stays deliberately under the container's
4Gi memory_limit.
Regenerated releases/app-catalog.json (still unsigned).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ga6N8Jk1YdCTMMX1LjDpAr
apps/cuprate/manifest.yml only ever wrote network/target_max_memory/
rpc.restricted.enable into Cuprated.toml, so the [tracing.stdout] and
[tracing.file] tables were silently absent — cuprated still applied
its built-in info/debug/7 defaults, but nothing on disk showed it.
Verified live on amishparadise 2026-09-01: the deployed 5-line file
had no [tracing] section at all, and the level was only discoverable
by running `cuprated --generate-config` and diffing.
Add both tables to the manifest's files[].content with the same
values cuprated already defaults to, so every new install ships a
Cuprated.toml an operator can actually read and tune. overwrite:false
means already-deployed nodes (amishparadise included) keep their
existing file untouched — this only changes what fresh installs get.
Regenerated releases/app-catalog.json (unsigned) to embed the updated
manifest; needs scripts/sign-catalog.sh before it's authoritative for
the fleet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ga6N8Jk1YdCTMMX1LjDpAr
A Claude worktree under aiui/ was committed as a submodule gitlink
(160000) with no .gitmodules entry, so actions/checkout's recursive
submodule pass exited 128 and every 'Build & push demo images' run since
v1.8.6 failed. Removing the index entry — the worktree is local state
and was never meant to be tracked.
Upstream publishes no docker images; our v0.18.4 image was built in-house.
This pass: official v0.21.2-beta release binaries (sha256-verified against
the signed release manifest), static, on alpine with the same entrypoint
shape as our existing image, pushed to our registry and smoke-run.
LND 0.21 auto-migrates the channel DB on first start (keeping a backup) —
the Update button is user-initiated, never auto-applied.
First upstream sweep since v1.8.5: the safe patch/minor pins, mirrored
into our registry first (source.archipelago-foundation.org/lfg2025/*).
Held for their own careful passes: the majors (grafana 11, nextcloud,
uptime-kuma 2, bitcoin-core 29, the DBs) and consensus-sensitive apps
(fedimint, electrumx 2.0). LND 0.21.2 needs an in-house image build —
upstream publishes none.
ssmithx
archipelago