Stage Cargo.lock and the regenerated public catalog in the release commit,
and re-fold AIUI into web/dist after the frontend build wipes it. All three
were manual catches on every one of the last three releases: the dirty
files failed the ISO preflight, the missing AIUI failed verify-artifacts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Branch stays alive for continued UX work; fast-forwarded to main after this
merge so it continues from the current tip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate answered a dead app with the bare string "app is not responding",
which on the app's own port reads as the node itself being broken — reported
against Gitea on a fleet node (the actual fault was a ghost container holding
Gitea's LevelDB lock, crash-looping the managed container). Serve the same
styled page as the login challenge instead: names the app, says it may be
restarting, and retries via a Refresh header (page CSP allows no script).
Status stays 502 so machine clients still see an upstream failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pasta exclusion paired with the transient-scope machinery: daemon-started
pasta/conmon inherit the starter's cgroup, so scopes kept app networking
alive across daemon restarts. A quadlet unit provides the same independence
(user.slice parentage) with supervision on top — Restart=always and
RestartSec=10, which also spaces restarts past pasta's port teardown (the
race observed live on strfry, 2026-08-10). Renderer needed nothing:
Network=pasta, ports and RestartSec were already in place.
Lifted in all four places: migrate_to_quadlet_if_needed, install_fresh, and
the missing-container unit-start path; the scoped start/restart helpers are
now unit-aware — quadlet-managed pasta routes through systemctl --user
(bare podman would fight systemd over an --rm container), legacy pasta
keeps its scope, so mixed fleets stay coherent during rollout.
Container suite 221/221.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-08-08 unwire set the bar: a DURABLE record of 'this app is
installed' must drive reaping, never runtime inference. installed-apps.json
is that record (written on install, cleared on uninstall, backfilled from
live containers, held through container absence). The reconciler's
companion loop now reaps against it — and only when the registry file
actually exists and parses: 'I could not look' and 'nothing is installed'
both surface as an empty set from the lossy loader, so a new
load_installed_apps_if_recorded keeps the distinction alive. The
runtime-derived set is deliberately not a fallback; it is the input class
that caused the incident. ORPHAN_GRACE still applies on top.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signed catalog has homed grafana at lfg2025/grafana:10.2.0 in the fleet
registry all along; the manifest still pulled Docker Hub's grafana/grafana,
so the repo-mismatch guard (correctly) discarded the catalog image on every
tick and warned every ~75s on every node. Registry verified to carry the
image; aligning the repo path makes the already-signed catalog entry
effective — no re-signing needed, no behavioural change beyond pull source.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turning discovery on prompts for a name; it rides the public announcement
(clean_display_name both directions: single line, control-stripped, 32-char
cap — it round-trips through untrusted relays). Blank lists as npub only;
off/on keeps the stored name; sending an empty name clears it. Discovery
lists show the name with the npub beneath. Own-npub display switches to
middle-ellipsis so the comparable tail stays visible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Re-keying requires the authenticated system.factory-reset' is precise and
useless to an operator standing at the seed screen. Map it to what they can
do: sign in, or factory-reset from Settings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The presence event deliberately never contains the onion — the npub is
what's actually visible on the relays, and the UI previously showed neither
it nor any way to find it. Same container, same styling; the copy button
copies the npub.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Presence gets a NIP-40 expiration (48h) and a 12h re-publish heartbeat that
honours the runtime toggle (UI-enabled nodes previously never re-published
at boot). discover() drops pre-TTL events client-side for relays that
ignore NIP-40. Switching discovery off publishes an empty tombstone, and
factory-reset tombstones BEFORE wiping identity — after the wipe the key is
gone and the stale event could never be replaced by anyone.
nostr.discovery-status now also returns the node's own npub (load-only).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The catch block dropped `loading` before branching, so every failed poll
unmounted the lock icon and status text for the 4s retry gap — the screen
flashed in and out on a live install test (zaza-optiplex, 2026-08-09).
`loading` now stays true across isServerStartingError retries; only success
or a genuine failure dismisses the waiting state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- The fit now scales screen X and Y independently (capped at 1.75x
anisotropy), so both 2D and 3D stretch to the container's aspect ratio —
a portrait phone uses its full height instead of shrinking the orbit to
the narrow width, and wide desktop panels spread horizontally.
- Legend/key centres at the top on mobile, mirroring the bottom-centre
2D/3D toggle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Compact containers (<480px) shrink the fit margins so the scene fills the
phone screen instead of floating in padding.
- 3D mode steepens its tilt on portrait (-0.95 vs -0.5) so the orbit reads
as a tall ellipse with depth, not a squashed horizontal band; projection
params are resolved per-aspect via modeParams() and re-resolve on resize.
- The 2D/3D toggle moves to bottom-centre on mobile (thumb reach, frees the
top edge); the hint line tucks above it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root causes of the flaky intro / janky revisits, all fixed structurally:
- Graph updates now RECONCILE against the live scene by identity (peer DID /
request id) instead of wiping the SVG: in-place updates for label/online/
trust changes, animated arrivals for new nodes, animated departures for
removed ones, and short-way-around slot glides when the ring layout shifts.
The old wipe-and-rebuild orphaned every in-flight intro tween whenever the
5s poll or the async self-DID fetch landed mid-intro — that was the
'sometimes parts don't show'.
- The intro is gated on readiness: first real container measurement (post-
layout, via ResizeObserver) AND the self node being present, with a 1.2s
fallback. Graph changes during the intro queue and apply on completion.
- Layout is deterministic per node: peers sort by DID for slot assignment and
jitter/phase hash from the DID, so backend response order can't shuffle
the map between polls.
- The long cinematic intro plays once per browser session; revisits get a
shorter, lighter entrance that doesn't fight the route transition.
- selfDid seeds from the cached localStorage value (re-cached on fetch) so
the centre node and links exist from the first frame.
- Requests' waiting-room orbit pushed clearly outside the peer rings (+1.0)
with loose per-request hover jitter — outside until accepted, then the
join animation glides them in.
- Full unmount hygiene: every tween killed by target (incl. the repeating
sonar pulse), fallback timer cleared, window listeners removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Inbound pending requests render as blinking yellow point-globes on an
orbit outside the peers, dotted-linked to self, with their own dashed
guide ring and a Request legend chip.
- Tapping one opens a black-glass popover (name, request message,
Accept/Reject) centred over the scene; tap-away or ✕ dismisses.
- Reject: the node swells and pops out of existence, taking its link.
- Accept: green burst ring, the point cloud/link morph to the trusted
colour and the globe glides inward onto the peer orbit, then the data
refresh replaces it with the real peer node.
- Wired to the same approve/reject RPCs as the pending panel; new
motionTokens.color.pending (#facc15) token.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Small glass 2D/3D toggle top-right of the map: tweens tilt/perspective/fit
(0.9s inOut) so the scene folds flat into the original radial 2D layout and
back. Default: portrait/mobile opens 2D, desktop 3D; last choice saved to
localStorage.
- Fix: setPointerCapture retargeted pointerup to the container, suppressing
click synthesis on children — node taps and the toggle never fired. Drag
now tracks via window listeners, no capture.
- Fix: stale post-drag distance made the click-suppressor swallow toggle taps
indefinitely ('stuck' toggle). Distance resets on every pointerdown and the
suppressor is one-shot.
- Map no longer slides under the floating mobile back pill: the
mobile-scroll-pad-back panel keeps its full 64px clearance when filled.
- Mobile DID copy/rotate card moved out of the header to below the view tabs
(new DidCardMobile.vue) and hidden on the Network Map tab.
- Node labels: dark stroke halo removed; 10px on mobile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Portrait containers now render near top-down and near-orthographic (tilt
-1.35, persp 16) so the map reads like the previous 2D radial layout —
rings become circles with no perspective squash. Desktop keeps the 3D
depth camera. Globe point-spheres unchanged on both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The map view turns the root into a flex column; default stretch alignment
was pulling the mode-switcher full-width on desktop. md:self-start restores
the content-fit pill (mobile stays full-width by design).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Nodes are now spheres made of points: fibonacci point-cloud globes with
depth-shaded dots, limb outline, and a slow local spin (drag adds parallax).
Self node is black — dark dots over a soft light backing disc with the
brand-orange sonar pulse marking 'you'. Spheres and dots sized down.
- Scene auto-fits and centres to the container on every device: the outermost
orbit is sampled through the real camera projection to get true bounds,
then scaled/centred between the overlays. Portrait screens tilt the camera
towards top-down so the orbit uses the full height (mobile readability).
- No idle orbiting: nodes hold position with a gentle side-to-side sway;
drag inertia now settles to a stop. Intro dolly pushes in (0.82→1) instead
of pulling back, so nothing clips during the intro.
- Labels get a dark paint-order halo and bump to 12.5px on mobile.
- Point clouds update at half frame rate to keep mobile/companion smooth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add gsap 3.15 + design-system-aware motion module (src/utils/motion.ts):
shared colour/duration/ease tokens mirrored from style.css, gsap.defaults,
live prefers-reduced-motion check.
- Replace the d3 force NetworkMap with NetworkMap3D.vue: peers on projected
3D orbital rings around the self node, cinematic intro (camera dolly +
staggered fly-in + ring draw), idle rotation with drag-to-orbit inertia,
depth-sorted painter's order, trust-colour palette, online/offline states,
sonar pulse on self, tap-a-node opens the detail modal.
- Map view now fills the dashboard panel to the bottom edge on desktop,
mobile and companion: .dashboard-scroll-panel:has(.node-map-stage) turns
the panel into a column (tab-bar/safe-area/audio-player aware padding)
instead of leaving the old dead bottom margin.
- Reduced motion: intro/idle skipped, scene renders static.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signed release catalog was pinned in 6c41852c; the curation source still
said :latest, which ec8d88a6 established can stall the fleet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The release tarball bakes demo/aiui in for fresh installs; the previous dist
predated the phase-13 security waves (S1-S7, W1.4) and would have shipped a
pre-hardening bundle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Quadlet renders --rm, so every stop DELETES the container: a scan during
the stop->start window legitimately sees nothing where an installed app
lives. The state layer treated that as the app not existing — after three
absent scans it removed the entry entirely, so dashboard tiles vanished
mid-restart, gate waits read 'absent' (grafana, at load ~2, ruling out
contention), and under daemon-restart churn the FIRST scan could publish
a completely empty My Apps map (observed twice on 2026-08-09).
Two guards, both keyed to the durable installed-apps.json registry:
- Eviction: an id the registry says is installed, and the user has not
uninstalled, is held as Stopped (health/exit cleared) instead of being
removed. The next scan that sees the container restores live state;
desired-state recovery still recreates genuinely lost containers; a
deliberate uninstall clears the registry first and still disappears.
- Empty scans: the first-scan exemption no longer applies when the
registry is non-empty — better to keep saying "scanning" than "empty".
Verified: scan-merge/absence/registry test filters 34/34.
Also carries the v1.7.127-alpha changelog (user-benefit entries for the
Tor self-heal, trusted invites, the three-layer mempool fix, vanishing
apps, the Bitcoin version pin, and the smaller UI fixes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An OTA updates the binary and web assets — not /etc/nginx, and not
/opt/archipelago/scripts. Auditing the 2026-08-09 fixes' delivery paths
found two that would silently reach nobody already installed:
- scripts/tor-helper.sh (reset-failed + truthful restart result) shipped
only via ISO builds and manual deploys. Now embedded via include_str!
like the doctor script, staged and installed at boot when the on-disk
copy differs.
- The /app/mempool/ nginx Upgrade/Connection headers existed only in repo
snippet sources consumed at image build time. A boot repair now
idempotently patches any mempool location block missing them — in the
live vhost (archipelago-http, the one sites-enabled actually links to),
the legacy file, and the installed snippet — and reloads nginx once.
Without this, every fleet node's mempool UI keeps loading-but-never-
connecting after the OTA that supposedly fixed it.
Both are non-fatal boot repairs in the existing bootstrap chain, no-ops
when everything is already current.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sticky tab header painted bg-black/70 across the modal — reported as
a black container where the design is transparent glass. The backdrop
blur alone keeps pinned tabs legible over rows scrolling beneath; the
tint drops to the standard bg-white/5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The gate accepted a client's websocket upgrade (.with_upgrades() on the
serving side) but proxy_to_app forwarded every request with a plain
hyper::Client, which completes the app's 101 handshake and then drops the
upgraded connection. Result: any ws-driven app behind a gated port loads
its page fine and then dies with close code 1006 on every connect —
mempool's entire UI is such an app, and the operator's browser console
was the only place the failure was visible. This was the THIRD layer of
the same outage (electrumx sync, then the nginx proxy path, now the app
tile's gated-port path — each fix exposed the next).
On an Upgrade request the gate now takes the client's OnUpgrade handle
from the request extensions, hands the handshake to the app with the same
sanitized headers (cookie/authorization stripping applies unchanged), and
on a 101 bridges the two upgraded connections with copy_bidirectional.
Non-101 upstream answers pass through so auth failures stay visible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard's /app/mempool/ proxy forwarded plain HTTP only: no
Upgrade/Connection headers, so the browser's /api/v1/ws handshake reached
mempool-api as a bare GET and Express 404'd it. The mempool page loaded
fine and every REST probe was green while the user saw a UI that never
connects — the backend was fully healthy and completely unreachable in
the only way that matters to the page.
This hid behind the electrumx initial-sync outage: once sync finished and
the API answered, "mempool works" was declared from REST checks while the
websocket path stayed dead. Reported by the operator from the browser,
which is the only place it was visible.
Adds Upgrade/$http_upgrade + Connection "upgrade" to the /app/mempool/
location in both shipped sources (image-recipe snippet + scripts copy).
Live-verified on the dev node: ws through the proxy now answers 101; the
control probe without upgrade headers reproduces the 404 signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full project build (vue-tsc -b, which also checks test files) caught
two default-ServerInfo constructors the scoped typecheck missed. Rather
than adding the field to both, the type now tells the truth for a
mixed-version fleet: daemons older than 2026-08-09 omit the field, so
consumers must treat absent as "unknown" and only === true as connected —
which is exactly how Home.vue already reads it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Home's Network card computed "Connected" from server-info's tor-address —
a string read from the hidden-service hostname file, which persists on
disk however dead the daemon is. Three fleet nodes ran with Tor down for
days while their dashboards said Connected; that indicator is why nobody
noticed. Server.vue's Tor label had the same flaw one step removed,
inferring "running" from services having onion_address values while the
correct signal (torDaemonRunning, backed by a real socket probe) sat
computed and unused twelve lines above.
Both now read liveness signals only: Home reads the new tor-running field
(server-side connect to 127.0.0.1:9050), Server.vue uses
torDaemonRunning. The address remains displayed as what it is — an
address — never as proof of life.
Verified: vue-tsc clean; rpc-client suite 79/79.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three of five reachable fleet nodes had Tor completely dead — Home said
"Connected" on all of them — and shipping the generator fix alone would
have repaired none of them. Two additions close that loop.
heal_on_boot (wired into the bootstrap repair chain): regenerate torrc
from current config with the fixed generator, and apply-and-restart ONLY
if the live file drifted or Tor is not answering on 9050. A healthy node
is left untouched. Without this, regenerate_torrc runs only from the Tor
RPC handlers and package install, so a node carrying a poisoned torrc
keeps it until someone happens to toggle a Tor setting — and worse, the
still-running OLD binary re-poisons on any such toggle: observed live on
the dev node at 07:20, when the running daemon rewrote torrc with the
unbindable gateway line hours after it had been hand-fixed. The heal
makes the fix self-applying on every restart, i.e. the OTA itself.
ServerInfo gains tor-running, populated by a real connect to
127.0.0.1:9050 each state refresh. tor-address is read from the
hidden-service hostname file, which OUTLIVES a dead daemon — it is a
configuration artifact, and the dashboard treating it as liveness is
precisely why three dead nodes showed "Connected" for days. Liveness now
comes only from the probe; the address stays a separate fact.
The heal reports the truth: it re-probes after restarting and warns if
Tor still is not answering, rather than assuming success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No node in the fleet could mint a Trusted federation invite or promote a
peer to Trusted from the dashboard. The cause is one case comparison.
Granting Trusted requires the node password; when none is supplied the
handler bails with "PASSWORD_REQUIRED: node password required to grant
Trusted" — a sentinel the frontend machine-reads (isPasswordRequired
checks includes('PASSWORD_REQUIRED')) to know it should open the password
prompt. sanitize_error_message allowlists "Password", but starts_with is
case-sensitive, so the ALL-CAPS sentinel failed the allowlist and
collapsed into "Operation failed. Check server logs for details." The
frontend never saw the sentinel, the prompt never opened, and the flow
died with a generic error on every node.
Reproduced via direct RPC: federation.invite {trust_level:"trusted"}
without a password returned the generic message; the same call WITH the
password minted a code fine — the machinery was healthy, only the signal
was being eaten.
"Tor address not available. Tor may not be running." — the invite
handler's other user-actionable precondition — was masked the same way
and is allowlisted too.
Regression tests pin both passing through verbatim, and
internal_errors_stay_generic still passes, so the generic masking of real
internals is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All ten AI data-access categories offer a context toggle, but only five
(Apps, System, Network, Media, Bitcoin) have any tool behind them — Wallet,
Files, Search, Local AI and Notes grant the assistant visibility with
nothing to act on. The Settings copy implied actions that do not exist.
Operator decision (2026-08-09): ship honest copy now, build the tools as
ordinary backlog. Each of the five carries a contextOnly flag, a "context
only" badge, and an appended description line saying the assistant can see
this but can't take actions with it yet. Drop the flag per category as
real tools land.
Verified: vue-tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two problems with running the gate against a mid-initial-sync electrumx:
1. The (now honest) protocol probe can only fail — ElectrumX serves no
sessions until it has caught up to its daemon, so the failure names a
state nobody can act on.
2. Worse, the destructive stop/start/restart tests actively destroy sync
progress: electrumx flushes its DB cache at 1GB, i.e. rarely, and every
restart discards all unflushed work back to the last flush. This node
spent 8d14h in initial sync largely because gate runs and reboots kept
taking hours of progress away — it restarted at 06:16 and resumed from
959,774, the same height it had reported hours earlier.
The suite now detects initial sync POSITIVELY — a fresh (<30 min)
"our height: N daemon: M" line from electrumx's own log, gap > 10 — and
skips the probe and the four destructive tests with the gap named:
# skip electrumx initial sync in progress (1672 blocks behind) — ...
This is not the container-absent skip trap fixed earlier: absence of the
log line means "unknown" and the tests run and fail honestly. Validated
against the live mid-sync node: all four guards fired with the real gap;
on a synced node the line shows gap 0-1 and everything runs.
Unblocks the release gate from waiting hours on a sync it was itself
prolonging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
changeTrust() wrote failures into `error`, which renders inside <NodeList>
— page content that sits BEHIND the open NodeDetailModal. So when
promoting a peer to Trusted failed, the operator saw the dropdown snap
back and nothing else: the error banner was painted on a surface the
modal was covering. Reported as "the error doesn't show in the modal",
2026-08-09.
NodeDetailModal gains an actionError prop rendered directly under the
Trust Level control, and Federation.vue routes failures there whenever
that node's modal is open (clearing on close and on the next attempt).
The page banner still serves failures with no modal in the way, and the
password-retry path keeps its own in-modal error as before.
Verified: vue-tsc clean; federation component tests 5/5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard's "Restart Tor" button dispatched to this helper and always
got {"ok":true} back. Two real defects, in order of importance — and one
disproved theory, recorded so nobody re-chases it:
1. reset-failed was missing. Once tor@default fails enough times systemd
latches "Start request repeated too quickly" and refuses to start it at
all; a plain restart is then a no-op no matter which unit you name.
All three fleet nodes found dead on 2026-08-09 were in exactly that
state, which is why the button appeared to do nothing.
2. The result was unconditional. The write-torrc branch waited up to 30s
for SOCKS and then ignored the outcome; the restart branch slept 3s and
claimed success. The UI reported "restarted" over a dead daemon.
Disproved: this was NOT wrong-unit targeting. `systemctl restart tor`
does propagate to tor@default — measured on austin-sapien, MainPID
changed. tor@default is still addressed explicitly because it is the
unit carrying the failed state worth resetting.
restart_tor_daemon() now: reset-failed, restart tor@default (fall back to
tor on single-instance installs), wait up to 30s for SOCKS on 9050, and
return {"ok":false,"error":...} pointing at journalctl when it never
comes up. Callers may no longer report success without a live SOCKS port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
regenerate_torrc widened SOCKS to the archy-net gateway whenever `podman
network inspect archy-net` reported one. Podman reporting a gateway is not
proof the host can bind it: under rootless podman the bridge and its gateway
live inside a network namespace and never appear on a host interface.
Tor binds listeners at STARTUP, not at config-check time — `--verify-config`
passes happily — so the failure lands as a refusal to start. austin-sapien,
2026-08-09:
[warn] Could not bind to 10.89.0.1:9050: Cannot assign requested address
[warn] Failed to parse/validate config: Failed to bind one of the listener ports.
[err] Reading config failed--see warnings above.
Tor then died completely — loopback SOCKS and every hidden service with it —
having run 3d 18h before something restarted it. The node's Home tab still
read "connected" because tor.service is Debian's multi-instance MASTER unit,
which is active(exited) by design; the real instance, tor@default, was failed.
Whether a node is affected is pure timing: archi-dev-box's torrc has only
`SocksPort 9050` because archy-net was not discoverable when its torrc was
last regenerated, so it fell into the loopback branch and Tor starts fine.
Nodes whose torrc was generated while archy-net was visible are one Tor
restart away from the same outage.
The gateway is now probed the way Tor will use it — bind an ephemeral port on
that exact address — and a failure falls through to the existing loopback-only
branch, which was already written and commented "fail closed". Widening SOCKS
must never be able to take the whole daemon down.
Verified on this host: 127.0.0.1 binds, 10.89.0.1 returns EADDRNOTAVAIL.
Tor restored on austin-sapien and confirmed end to end — bootstrapped 100%,
and a request through the proxy returns {"IsTor":true}.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both electrumx tests did socket.create_connection(("127.0.0.1", 50001)) and
called it proof the service was up. It is not: that is the HOST-published
port, and podman's port forwarder completes the TCP handshake even when
nothing inside the container is listening.
The cost was real. Both tests were green while mempool-api sat in a hard
ECONNREFUSED loop against electrumx:50001 from inside archy-net — the
operator's "mempool doesn't connect to bitcoin". Measured side by side:
127.0.0.1:50001 from the host OPEN, electrumx:50001 from a sibling container
REFUSED.
The cause is not a misconfiguration — mempool-api's CORE_RPC_HOST,
ELECTRUM_HOST and ports are all correct. electrumx has not finished its
initial sync (DB height 959,774 vs chain tip 961,706, ~1,932 blocks, 8d 14h
elapsed) and ElectrumX does not serve sessions until it has caught up.
Now both tests do a server.version round-trip and require a JSON result, so a
forwarder that accepts-and-drops fails. Verified against the live node: the
new probe FAILS today where the old one passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moves the default off 29.3.knots20260508, which applies the BIP110/RDTS
network upgrade and HALTS until an operator sets consensusrules=rdts. A node
running it sits frozen — 100.64.204.114 is stuck at block 961,692 with blocks
AND headers static, 11 peers and no pruning, while reporting
initialblockdownload=false and verificationprogress 0.99998, i.e. "synced".
The nodes on 20260210 are at the tip.
The old catalog's default was {"version":"latest", image: ...20260508} while
the manifest's :latest tag resolves to 20260210 in the registry, so a fresh
install and a plain pull produced different daemons. That is almost certainly
how one node ended up on the RDTS build and its siblings did not.
20260508 stays selectable — adopting RDTS remains possible, just deliberate.
Diff against the previously-signed catalog is exactly one app, all four
fields: version, image, default and the embedded manifest. Registry hosts
unchanged and inside the trust floor; verified by
check-catalog-registry-trust.py and `ceremony verify`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sign-catalog.sh refused to sign a freshly regenerated catalog:
REFUSING: the catalog names registry hosts the deployed fleet does not
trust. source.archipelago-foundation.org — 78 image refs
It was right, and the check earned its place. apps/*/manifest.yml moved to
the public domain in 8e814ca0, but releases/registry-trust-floor.json still
lists only docker.io, ghcr.io, localhost and the OVH host — the migration is
ship-binary -> confirm-fleet -> promote-floor -> regenerate, and only the
first move happened. A full regen therefore produced a catalog naming a host
no deployed binary trusts, which would have failed every install in the field
with "not from a trusted registry". Verifying that an image PULLS from the
domain (it does, including from a fleet node) does not verify that shipped
binaries ACCEPT it — different checks.
Two fixes, no manifest churn:
- image-versions.sh honours a caller-supplied ARCHY_REGISTRY instead of
overwriting it, so generation can target a host inside the floor.
- the generator retargets OUR registry host inside embedded manifests to
whatever it is generating against, leaving docker.io/ghcr.io refs alone.
REGISTRY moves above the embed block, which now depends on it.
The repo keeps naming the public domain; only the generated artifact is
pinned to what the fleet can actually use. When the floor is promoted,
generating against the domain becomes a no-op here.
Verified: check-catalog-registry-trust.py passes, 11 docker.io/ghcr.io refs
untouched, and the diff against the signed catalog is exactly one app —
bitcoin-knots, :latest/20260508 -> :29.3.knots20260210 across all four fields.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalog's default version was {"version": "latest", image:
bitcoin-knots:29.3.knots20260508, default: true}, so a fresh install — or
anyone selecting "latest" — silently got the BIP110/RDTS build. Meanwhile
the manifest's own :latest tag resolves to 29.3.knots20260210 in the
registry, so the two paths installed different daemons.
That matters because 20260508 does not merely enable RDTS, it HALTS until
an operator sets consensusrules=rdts. Node 100.64.204.114 runs it and is
frozen at block 961,692 — blocks AND headers static across 30s, 11 peers,
unpruned — while reporting initialblockdownload=false and
verificationprogress 0.99998, i.e. "synced". The two nodes on 20260210 sit
at the tip, 961,706. Re-pointing a tag could have done that to every node
at once.
Pinned in all three places that would otherwise re-introduce a moving tag:
the manifest, image-versions.sh, and the catalog VERSIONS table. The
"latest" pseudo-version is removed outright — a moving default is not
appropriate for a consensus-critical app — and the default is now the
explicit non-RDTS 29.3.knots20260210. 20260508 remains selectable, so
adopting RDTS stays possible but becomes deliberate.
Verified: both tags' daemons read via `bitcoind --version` — :latest and
:29.3.knots20260210 are both v29.3.knots20260210, so this pin is
behaviour-neutral on nodes already running latest.
NOT DONE HERE: releases/app-catalog.json still carries the old signed
entry. It must be regenerated and re-signed by the operator before it
takes effect — catalog entries override on-disk manifests (origin-wins).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two fixes from the first full 125-test run (121 ok / 4 not ok).
1. The Quadlet unit test still asserted Restart=on-failure for backends,
which 8908fb4f deliberately changed. All 32 units on the node are
Restart=always; only archy-btcpay-db showed up because `fail` aborts at
the first offender. The old rationale — "backends need on-failure so an
operator `systemctl stop` stays stopped" — was wrong twice over:
systemd never applies Restart= to a unit stopped via `systemctl stop`,
and because quadlet renders --rm, a cleanly-exited container is deleted
and on-failure never brings it back (bitcoind exits 0 on SIGTERM, so
backends vanished). Verified on-device before that change: `podman stop
bitcoin-knots` came back in 12s, a dashboard stop stayed stopped 90s.
Now asserts Restart=always, with the corrected reasoning in-place.
2. all-apps-lifecycle cycled ten apps back-to-back and manufactured the
contention that then failed later tests: load went 10.8 at preflight to
17.6 mid-loop, and btcpay's recovery stretched from 52s on a quiet box
to 216s and then 512s, defeating waits of 180s and 300s. Widening waits
cannot win when the run sets the load they must survive, so the loop now
drains between apps — waits for load1 under 2x nproc, capped at 90s.
ARCHY_APP_SETTLE_SECS=0 disables; ARCHY_APP_SETTLE_LOAD overrides.
Verified: run.sh use-quadlet-backends-install → 6/6 (was 1 failure).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both failed a gate run on a healthy node.
bitcoin-receive already tolerated WALLET_LOCKED, but its 180s window
starts when the TEST starts and the lnd restart that locks the wallet can
land partway in. On 2026-08-08 the restart hit 65s in and the wallet
unlocked at 2m25s (journal: lnd.service started 20:11:05, "wallet has
been unlocked without a time limit" 20:13:48) — 48s after the deadline
expired. The daemon's own unlock budget is ~10 min because opening the
channel and graph dbs takes minutes on a loaded box, so 180s was stricter
than the thing under test. Now 420s, ARCHY_LND_UNLOCK_SECS.
btcpay's start wait was 180s, but stopping btcpay DELETES the container
(quadlet renders --rm), so package.start is a full dotnet recreate rather
than a container start. Measured 52s on a quiet box; it exceeded 180s
during a gate run on the same node at load ~11. Now 300s,
ARCHY_BTCPAY_START_SECS.
Neither change masks a lifecycle fault: both paths were verified by hand
to complete correctly, just slower than the assertion allowed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stop/start/restart loop carefully accumulates per-app failures into
$fails and prints them, but the three rpc_result calls were bare. Under
bats' errexit a bare call ends the test immediately, so the summary that
names the app never ran.
On 2026-08-08 that turned a single transient error into an unattributable
failure: the test died at package.stop with no indication which of the
ten targets was involved. It was mempool, and the identical call returned
{"status":"stopping"} by hand a few minutes later.
Each call now records <id>:<phase>-rpc and moves to the next app, so the
run reports what actually broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ui-coverage.bats and ui-probes.bash hardcoded https:// while the rest of
the harness builds URLs from ARCHY_SCHEME via lib/rpc.bash. That made the
suite unrunnable on a node serving the dashboard over http.
On archi-dev-box :443 is bound to the Tailscale / WireGuard / LAN
interface addresses but NOT to loopback, while :80 is bound on 0.0.0.0.
So five probes failed with "curl failed (network/timeout)" against
endpoints that were serving 200 the whole time — http://127.0.0.1/,
/catalog.json, /app/lnd/, /app/electrumx/ and /app/mempool/ all verified
200 by hand.
Default stays https, so nodes that already exercise the TLS path keep
doing so. Test titles drop the hardcoded scheme, since they no longer
describe which one ran.
Verified: ARCHY_SCHEME=http ./run.sh ui-coverage → 9/9 (1 skip),
previously 5 failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gate run on a box at load ~14 failed five times over, and every failure
read "could not create a container" — searxng:start, package.start
btcpay-server, 3x electrumx — never a lifecycle fault. That sent two
separate sessions hunting a phantom host-wide cgroup failure. It was
contention.
Measured on the 4-core node: at load ~14 podman runs 9-16 processes deep
and healthchecks time out 3-8/min; at load ~3.7, podman ~1 and zero
timeouts. Restoring four containers whose HealthTimeout equals their
HealthInterval, unchanged, made the box *better* once load fell — so the
config is a latent hazard, not the cause here.
Preflight now checks, once, before iteration 1:
- aardvark-dns is singular (duplicates desync name resolution)
- load1 is under nproc+1, waiting up to 15 min for a spike to pass
- podman can actually create a container, 3/3
Deliberately NOT checked: the count of "Failed to create container" in
the journal. Those lines come from healthcheck exec churn and post-boot
settling, never reach 0 on a busy node, and gating on them would block
the gate forever. The probe proves creation positively instead.
Escape hatches: ARCHY_PREFLIGHT=0, ARCHY_MAX_LOAD, ARCHY_PREFLIGHT_SECS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An app whose container disappeared could be lost permanently. Desired-state
recovery decided whether to recreate it from running-containers.json — "what
was running at the last snapshot" — which is a different question and a
perishable answer: it records only what is running NOW, so an app that stays
down long enough simply ages out. Once out, boot's ExistingOnly mode will not
recreate it, because it cannot tell "installed and lost" from "merely
available in the catalog". Manifest still on disk, nothing to bring it back.
This is the second occurrence of one root cause. indeedhub-minio/-postgres
went permanently absent on one node (2026-08-06); the fix then was
`absent_stack_member_with_live_sibling`, which only rescues a stack member
that still has a living sibling. bitcoin-knots is standalone, so on
archi-dev-box (2026-08-08) it vanished, aged out, and stayed gone — LND
crash-looping on `lookup bitcoin-knots: no such host` for hours, electrumx
unable to reach its daemon, and an orphaned fedimint container waiting 30
hours for a host that no longer resolved. Recovering it took a manual
reinstall. This is the general fix the narrow one implied.
Installation is a DECISION, not a runtime observation, so it gets a record
that no amount of downtime erodes: installed-apps.json, written when an
install succeeds and cleared on uninstall, in the same breath as
mark_user_uninstalled — leaving a stale claim would let recovery recreate the
app that was just removed. It is the durable counterpart to the
user-uninstalled marker that already existed.
Safety, in order of how badly each could go wrong:
- Cannot resurrect a deliberate uninstall: user_uninstalled is checked
earlier in ensure_running_with_mode and returns before anything is created,
and uninstall clears this record too.
- Cannot install an app nobody asked for: only names in the record qualify,
and ExistingOnly's other guards are untouched.
- Cannot mislead a node upgrading into the feature: backfill seeds from
ADOPTED containers only — evidence that something is really there — skips
anything user-uninstalled, is additive so a momentarily-down app is never
dropped, and no-ops on an empty adoption list (podman unreachable must not
read as "nothing is installed").
Four tests, including the one that states the point: the record must outlive
a running-snapshot that has gone empty.
Container suite 221/221, crash_recovery 15/15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two operator reports on the same screen.
The section had no card. Every other Settings section wraps itself in
`glass-card px-6 py-6 mb-6` — AccountSection, AIDataAccessSection,
NodeCertificateSection, BackupSection, the lot — and this one rendered as
bare text on the page. Reported twice, because the wrapper lives in the new
component and nothing about adding `<LightningCredentialsSection />` to
SystemSection.vue's list tells you it is missing. Heading moved to h2/text-xl
to match its siblings. A test now asserts the card, so a third report is not
needed.
And rotating told the operator Lightning did not exist. Rotation restarts
LND, so `status.installed` reads false for a moment — and the template read
that literally: "Lightning is not set up on this node yet, so there are no
credentials to rotate. Install the Lightning app first." Seconds after
rotating. On a node with a working wallet. It also replaced the progress they
had every reason to be watching, on the one action that invalidates every
credential their wallet holds.
A container briefly absent is what rotating LOOKS like, not evidence
Lightning was never there. The not-installed message is now gated on
`!rotationInFlight`, which covers both `running: true` and the awaitUntil
window between asking for a rotation and the node reporting one — `installed`
can already be false in that gap, so gating on `running` alone would have
left the same hole. Mid-rotation with no status yet says "Rotating
credentials — Lightning is restarting" instead of falling through to a
details block with empty fields.
awaitUntil became a ref so the computed re-evaluates rather than holding a
stale value until some other reactive dependency happens to change.
Three tests: the card exists; a running rotation does not claim Lightning is
missing; and — the half that matters just as much — a node with genuinely no
Lightning still gets told there is nothing to rotate, so the fix has not
simply hidden a true statement. 16/16, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reproduced live: the operator pressed install on ElectrumX shortly after
Bitcoin Knots started and got
Bitcoin RPC returned 500 Internal Server Error while checking pruning
status: {"error":{"code":-28,"message":"Verifying blocks…"}}
-28 is RPC_IN_WARMUP — bitcoind's normal path on every start, lasting
minutes on a large chainstate. check_bitcoin_pruning_compatibility retried
`for _ in 0..3` with a 2s sleep, so it gave the node about six seconds and
then reported warm-up as a hard failure. Any app requiring unpruned Bitcoin
was therefore uninstallable in the ordinary window after Bitcoin starts.
This is the likely mechanism behind the operator's "fedimint gateway
disappeared at 88% install": both fedimint apps declare a bitcoin-core
dependency and sit in exactly that window.
The same install path already knew better. wait_for_bitcoin_rpc_gate waits
180s precisely because getblockchaininfo answers during sync. This check
runs earlier and disagreed — one concern, two contradictory answers, in one
install. It now shares the budget, and a test asserts the two stay equal.
Only NOT-READY is waited out: -28 by code, plus the "loading block index" /
"verifying blocks" / "rewinding blocks" message shapes for a proxy that
rewrites the envelope. A genuine fault — bad auth, method not found,
unparseable body — still ends the loop on the first response, so a broken
RPC fails fast instead of burning 180s. Both directions are pinned by tests,
because being too loose here is as bad as being too strict.
Warm-up is announced ONCE to the install log, so a slow install reads as
"waiting for Bitcoin" rather than a stall. And the failure message now says
what to do — "Bitcoin is still starting up… wait until it reports it's
synced, then try again" — instead of pasting the raw JSON-RPC envelope,
which was accurate and useless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caught on archi-dev-box within minutes of deploying 3ac59a73: the reaper
removed archy-bitcoin-ui and archy-lnd-ui, whose backends ARE installed.
archy-bitcoin-ui was gone for 36 minutes, until the operator reinstalled
bitcoin-knots and `reconcile` put the companion back.
Not a logic error — the arithmetic did what it was told. The inputs were
false. Both backends' containers were missing because of the clean-exit
vanishing bug (8908fb4f), and both had already aged out of
running-containers.json, which only ever records what is CURRENTLY RUNNING.
So the two signals `installed_app_ids` combines are not independent: one root
cause falsifies both simultaneously. ORPHAN_GRACE could not help either — the
condition was persistent, not transient, which is exactly the case the grace
period cannot distinguish.
The asymmetry decides it. An un-reaped orphan costs a stale UI tile. A
wrongly-reaped companion costs a working screen and turns one lost app into
two — the reaper amplifies the very failure it was meant to tidy up after.
`reap_orphans` and its tests stay, documented as NOT TO BE WIRED until a
durable record of "this app is installed" exists to drive it. Inferring
installation from runtime state cannot answer that question, however many
runtime signals are combined.
The provisioning half is untouched and is the actual fix for "fedimint
installs but does not work": driving `reconcile` from installed_app_ids means
a companion is never stood up for an app nobody installed, so no NEW orphans
appear. The one genuine orphan on this node (archy-fedimint-ui, for an app
never installed) was correctly removed before this change landed.
Unit tests passed the reaper because they verify the set arithmetic, not
whether the "installed" signal is truthful. Only the device could show that.
Container suite 221/221.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`sign_nip98` read the node's pubkey through `get_nostr_pubkey`, which goes
through `load_or_create_nostr_keys`. On a node with no Nostr identity that
does not fail — it GENERATES a keypair, writes the secret to disk, and signs
with it. So an IndeeHub auth header could quietly create a new node identity
as a side effect, then authenticate as a stranger holding a key nobody has
ever seen. The `.context("node has no Nostr identity")` guarding the call
could never fire, because the call could never fail that way.
`nostr_identity_exists` is the missing distinction: bootstrap may create,
but anything AUTHENTICATING as this node must prove the identity it already
has. sign_nip98 now gates on it and bails loudly.
Caught by `a_nip98_event_names_the_exact_url_and_method`, which asserts
exactly this ("must fail loudly rather than sign something empty") and had
been failing since the file landed in 58c759c1 — invisible because the
earlier runs on this branch filtered to `container::`.
Full bin suite 1381/1381.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.
Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.
It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.
Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.
Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.
Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AIUI has carried 353/356 as a known-open item (W1.7) for long enough that
the three reds were treated as background noise. All three were the tests
being wrong. 357/357 now.
`injects web results into system prompt when enabled` asserted
`body.webSearch === true`. The code deliberately sends false there:
`proxyWebSearch = webSearchEnabled && !clientSearchSucceeded`, so when the
client-side search has already run and injected its results into the system
prompt, asking the proxy to search again would be a second redundant search
on every turn. The assertion predates that change. Fixed, and the other half
of the contract added as its own case — zero client results must still ask
the proxy to search.
The two seed-songs failures ("extracts 10 songs", and the conversation
regression built on the same fixture) were one wrong number:
`expected: { songs: 10 }` against an assistantResponse containing exactly
six `song_ext` entries and ending coherently on Treefingers. Not truncated —
just miscounted. The extractor was returning the right answer the whole
time. Counted from the fixture rather than from intent, so the number now
describes the input instead of contradicting it.
Full branch state after the main merge: AIUI 357/357, neode-ui 972/972
across 114 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e36f36ee re-encoded the chat backdrop to webp (1052K -> 478K) to fix "the
background takes ages to load", but left the JPG in place. ChatPage.vue
references only `bg-intro-3.webp`, so the 1053K JPG has been shipping in
every AIUI bundle, every deploy and every ISO since, referenced by nothing.
Checked before removing, because the same filename IS live elsewhere:
`appgate/mod.rs` serves `bg-intro-3.jpg` as one of four LOGIN_BACKGROUNDS,
but from `/opt/archipelago/web-ui/assets/img` — the neode-ui copy, which is
untouched here. Only AIUI's duplicate goes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reported: "`files` context request times out". sanitizeFiles makes three
sequential calls into the File Browser app — login, getUsage, listDirectory
— wrapped in a try/catch. A catch only sees a REJECTION. A socket that
connects and then says nothing leaves the promise pending forever, so
handleContextRequest never posts a `context:response` and the AIUI side sits
until its own bridge timeout instead. The File Browser is a plausible source
of exactly that: on this node `/app/filebrowser/api/resources/` does not even
route (404), and its session-cookie path is the subject of a separate open
bug.
The guard goes at handleContextRequest rather than inside sanitizeFiles, so
no category — present or future — can strand the bridge. `files` is merely
the one with three network hops today; sanitizeSystem is also async.
withTimeout resolves rather than rejects, because the caller's one job is to
always answer, and a rejection would just relocate the problem into a catch.
A late null is safe by the protocol's existing shape: the AIUI reader
already treats a response with no usable data as "nothing to show", the same
as an empty category.
Two tests: a never-settling File Browser still produces a
`context:response`, and a healthy category still returns real data rather
than being flattened to null. 27/27 contextBroker, vue-tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges gitea-ai/main (65 commits) into the phase-13 branch (419) so one
build carries both lines — the AIUI/assistant/container work and the
open-source readiness work (licensing, the marketplace DID signature layer,
the registry domain migration, the secrets and infrastructure scrub).
Every Rust file auto-merged. The container fixes from this branch and main's
registry-domain migration and node-name genericisation coexist without
manual intervention.
Conflict resolution — all of them were modify/delete, and all were resolved
in main's favour deliberately:
`.planning/**`, `scripts/deploy-to-target.sh` and `scripts/setup-aiui-server.sh`
were deleted by main's `6ba05996` ("security: remove all infrastructure and
internal process material from the repo") and added to .gitignore there.
Keeping this branch's copies would have re-committed internal process and
infrastructure material into a repo being prepared for publication, silently
undoing that cleanup. Resolved with `git rm --cached`, so every file remains
on disk locally and in this branch's history — it is untracked, not lost.
The remaining .planning files this branch added after the merge base were
untracked the same way, so the result is consistent rather than half-tracked.
Container suite 221/221 on the merged tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13-15's gate requires the DEPLOYED surface to be checked, not only the
source. That half is currently void: archi-dev-box runs a binary dated
2026-08-08 03:20 built from main, not this branch — no `app_uninstall` in
`strings`, and the ownership hooks chown unconditionally with no drift-gate
`stat` calls, so b9e64eb6/db8937f9/ca106c5a/b8869307 are all absent. Every
row of the acceptance table was verified against a binary the node no longer
runs, 417 commits back.
Also corrects row 2. The record captured scope `own` only, which cannot
discharge check 2's "real peer/owned files"; and the 2026-08-06 note saying
peers/owned "exist only in type signatures" is obsolete — 05b459a6 and
9abc1623 made requestArchyAllContent fetch all three scopes from init(),
deduped through one sink, which the operator confirmed on 2026-08-08 is the
intended auto-load-at-init behaviour. Check 2 is code-complete and
verification-pending, not unbuilt.
Records the ordered steps to actually close the phase, and the pre-deploy
orphan-companion evidence snapshot the reaper will consume.
The phase stays OPEN. Nothing here closes 13-15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archi-dev-box logged `reconcile failed app_id=btcpay-server error=chown
/var/lib/archipelago/postgres-btcpay failed with status exit status: 1`
while BTCPay was running and healthy and there was nothing to repair:
`find /var/lib/archipelago/postgres-btcpay ! -uid 100998` returns zero
files, and the identical command run by hand exits 0. The chown through
`sudo systemd-run` had simply failed once, and that transient failure
propagated out of the pre-start hook and took the app's entire reconcile
with it.
These hooks exist to repair OLD installs. On a healthy node the repair is
already a no-op, so its failure is not evidence of anything being wrong.
repair_dir_ownership folds the gate, the chown and the verdict into one
place: skip when ownership is already right, chown when it is not, and on a
failed chown RE-PROBE before deciding it matters. If the ownership is
correct anyway — a concurrent repair, or a transient sudo/systemd-run
failure on an already-correct tree — warn and continue. Only a chown that
fails AND leaves the ownership wrong is an error, which is the case the
loud failure was written for: a mis-owned volume the app genuinely cannot
open.
Replaces the three hand-rolled gate+chown+bail blocks in
ensure_btcpay_stack_dirs and the one in ensure_fedimint_dirs.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Writing the first tests for this section found the bug they were written to look
for. `rotate()` started the poll, then the `load()` immediately behind it took a
status snapshot that did not yet carry `running: true` and cancelled the interval
— so the screen froze on the one action that most needs to show progress. The
operator has just invalidated every credential their wallet holds, the rotation
is genuinely running on the node, and the page tells them nothing is happening
until they reload it by hand.
It survived manual review because the backend flips `running` inside the same
critical section that accepts the request, so the happy path usually wins the
race. "Usually wins a race" is not a property to ship on a credential rotation.
Polling now continues for a bounded window after a request the node accepted,
and stops early as soon as `running` is observed. Bounded, so a request that was
accepted but never acted on stops polling rather than hammering the node.
12 component tests cover the states that carry consequences: the channel census
shown before the button is offered, the stale-BTCPay warning, the difference
between "BTCPay has no internal node" (silence — an absence, not a fault) and
"BTCPay's credential is dead" (a warning), the block on rotating while LND is
unreachable, both poll races above, and that an idle tab does not wake the node.
Verified: 12/12 new, 880/880 frontend tests, vue-tsc clean, and the rebuilt
bundle contains the new strings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mint wait and the post-rotation verify shared one 15-minute budget. A
rotation that legitimately spent 14 of those minutes waiting for LND to mint a
fresh macaroon — normal on a loaded node, where opening channel.db/graph.db/
wallet.db alone has been measured at 2m38s — then had 60 seconds to confirm the
node identity and channel census came back, and would report FAILURE on a wallet
that was completely healthy.
That is the most alarming possible way to be wrong about someone's Lightning
node: it names a backup directory and tells them to investigate before retrying,
at the exact moment nothing is actually broken. Each wait now gets its own
budget. Waiting longer costs nothing here — the failure this step exists to catch
(changed identity, missing channels) is not time-sensitive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 2.
The `indeedhub` submodule points at a Gitea repo that is not public and carries
no known licence (the licence audit defers it: "partnership in place; license
the submodule before/at public release"). An outside developer running
`git clone --recursive` today either fails on auth or pulls unlicensed code —
a bad first five minutes with the project either way. It was never checked out
in this tree.
Removing it costs nothing, because nothing builds from it:
- `indeedhub-demo/Dockerfile` states in its own header "No submodule or local
source needed" and clones the public GitHub mirror instead.
- Every other `indeedhub/` reference in the tree is `apps/indeedhub/` — the app
package — which is a different path and untouched. The app itself ships as a
container image from the registry and is unaffected.
Kept `indeedhub-demo/` rather than dropping it as the plan suggested: it is a
working, self-contained demo build with no submodule dependency, which is
exactly the shape the rest of Phase 2 is moving toward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 2. Removed (~17 MB, 12.8k lines):
- `.githooks/pre-push` — the hook that re-committed the 27 MB companion APK on
every push, which the plan names as the root cause of the 5.5 GB history.
Verified inert first: `core.hooksPath` is unset, so it only ever ran for a dev
who opted in by hand.
- `neode-ui/dev-dist/` — generated vite-plugin-pwa output (a Workbox bundle),
tracked and not ignored. Added to .gitignore so it cannot come back.
- `Android/archipelago-0.3.0-debug.apk.zip` — 16 MB, stale, zero references.
- `RELEASE-NOTES-v1.0.0.md` — superseded by CHANGELOG.md.
- `docs/container-architecture.html` (311 KB) and the two generated archive
HTML artefacts, whose rows are removed from the archive index in the same
commit so the table doesn't point at deleted files.
THREE items the plan lists were verified and deliberately NOT deleted — the
plan is wrong about each, and following it literally would have lost content or
broken a build:
- `neode-ui/docs/GAMEPAD-NAV-MAP.md` is called "a duplicate of
docs/GAMEPAD-NAV.md". It is 660 lines against that file's 159 — four times the
content, not a copy. Needs a human read to decide what to keep.
- `Android/app/debug.keystore` is called "standard practice" to remove. This
repo deliberately commits it: `build.gradle.kts` sets
`storeFile = file("debug.keystore")` and `Android/.gitignore` carries an
explicit `!/app/debug.keystore`, with a comment explaining it exists so every
machine produces the same debug signing identity. Deleting it breaks Android
debug builds.
- The three "move to release assets" binaries are not a pure git operation —
two have live consumers. Detailed in the next message rather than guessed at.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Bitcoin Knots disappeared again, plus other apps." Root cause is a pairing,
not a single bug: quadlet renders `podman run ... --replace --rm`, so the
container is deleted the moment it stops, and from_manifest set
Restart=on-failure, which declines to restart after a CLEAN exit. bitcoind
exits 0 on SIGTERM. So any clean stop deleted the container AND left it
deleted — the app vanished from podman and from My Apps until a later
archipelago reconcile tick noticed and recreated it. That is the
"previously-running app has no container after boot — recreating
(desired-state recovery)" line, which fired for bitcoin-knots at 18:53,
19:57 and 20:39 and for electrumx at 19:57 and 20:42 on 2026-08-07.
A crash always self-healed: on-failure restarted the unit and podman run
recreated the container. Only a clean exit stranded it, which is why this
survived so long.
The justification for on-failure was wrong on systemd's own semantics. It
read "clean exits — e.g. operator-issued systemctl stop — stay stopped", but
Restart= is never consulted for a unit stopped via systemctl stop
(systemd.service(5)), and that is exactly how archipelago stops these apps
(prod_orchestrator -> stop_service_with_timeout). Always keeps the
stopped-stays-stopped behaviour and drops the failure mode.
Always also restores the premise of the Quadlet migration — systemd owns
supervision, so an app returns without archipelago alive to notice it left.
Checked before flipping: no manifest declares a one-shot container and there
is no manifest-level restart field, so nothing gets restart-looped.
Propagation to existing nodes is via sync_quadlet_unit's drift re-render,
which rewrites the unit and daemon-reloads WITHOUT restarting the service —
running containers are undisturbed and the new policy governs the next start.
OnFailure is kept as a deliberate opt-in with a note not to wire it back to
backends. Two tests now pin the new default and assert on-failure is absent
from a rendered backend unit.
Container suite 215/215.
NOTE FOR THE OPERATOR: this changes supervision semantics for every app on
the Quadlet canary path. Wants sign-off and a lifecycle-gate run before OTA.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
installed_app_ids judged installation on live containers alone. Watched on
archi-dev-box within the hour: lnd read as ABSENT, then as EXISTS again.
Containers on this node come and go — the boot reconciler logs
"previously-running app has no container after boot — recreating" for
bitcoin-knots and electrumx repeatedly — so a momentary gap looked exactly
like a removal, and the reaper would have taken a healthy companion's unit
with it. ORPHAN_GRACE narrows that window but cannot close it: nothing
bounds how long a gap lasts.
An app now counts as installed if its container exists in any state OR its
container name is in the durable last-running snapshot. That snapshot is
what crash_recovery itself calls "installation evidence" and what
reconcile_all_with_mode already trusts to recreate a previously-running app
whose container vanished — the same signal, for the same reason, now shared
rather than reinvented.
Only fedimint is a true orphan on this box: it appears in no adoption list
and has no quadlet unit of its own. lnd is installed and merely flapping,
which is a separate bug.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both default update mirrors resolve to the SAME host — the primary by name over
HTTPS, the fallback by IP over plain HTTP — while SystemUpdate.vue told the
operator "Servers this node checks for updates. The primary is tried first; if
it's slow or unreachable, the next one in the list is tried automatically."
That promises availability redundancy the pair cannot provide: if the origin is
down, both entries are down. Reported by the operator, who read the list and
correctly concluded the fallback made no sense.
The mechanism is fine and deliberate — it recovers a node whose DNS is broken
or whose clock is wrong, both of which fail TLS while plain HTTP still works,
and it is safe because the manifest carries an Ed25519 signature verified
against the pinned release-root anchor, so transport integrity is not what
protects the update. (That last part only became true once Workstream B pinned
the anchor; before then this fallback would have been a real hole.)
So the bug was the labelling, not the design:
- Backend label "Direct (fallback)" -> "Same server, no DNS/TLS", and the
comment now states plainly that it is the same host, what it recovers, and
that real redundancy needs a different one.
- UI copy now scopes the redundancy sentence to genuine mirrors and adds a
paragraph saying the two built-in entries are one server, what the second
actually recovers, that it does not help if the server is down, why an
unencrypted fetch is acceptable, and how to get real redundancy.
The relabel reaches existing nodes: force_ovh_update_primary rewrites labels for
the two default URLs on every load, while the merge matches on URL and never on
label — without that rewrite path a renamed default would have sat in the code
and never propagated to a single deployed node. Noted inline so it is not
re-broken.
Verified: 40/40 update tests pass (including the mirror load/merge/strip ones),
vue-tsc clean, build green, and the new copy is present in the freshly built
SystemUpdate chunk. Nothing in the tree pinned the old label string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Failed to scroll to index N after 10 attempts" appeared in the console on
every send. It was not a real failure — it was two scroll controllers
arguing.
scrollToBottom() called virtualizer.scrollToIndex(last) AND then assigned
el.scrollTop on the next tick. scrollToIndex runs a retry loop that nudges
scrollTop toward the target row's measured offset and re-checks, up to ten
times, because dynamically-measured rows move the target as they settle.
The manual assignment overwrote each nudge, so the loop never observed
itself converge and always exhausted its attempts.
For "go to the end" the index-settling machinery buys nothing: scrollHeight
already is the bottom, the virtualizer renders whatever window that offset
implies, and it keeps working while a response streams and the last row
grows — the case the manual fallback was added for in the first place.
scrollToMessageIndex still uses scrollToIndex, which is the right tool for
jumping to an arbitrary row.
Console-only change; needs a device check that the chat still pins to the
bottom while streaming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open-source readiness plan, Phase 1 items 3 and 5.
Item 3 turned out to be far narrower than the plan's "93 files" once each hit
was classified rather than bulk-replaced. Sanitized only genuine operator
identifiers:
- FIPS test fixtures and a pine_ha comment carried real node LAN addresses ->
RFC 5737 TEST-NET-1, the convention already used elsewhere in this repo.
- Real tailnet addresses in fips/endpoints.rs, mock-backend.js and the mesh
test runner -> the base of the CGNAT range, obviously synthetic.
- Incident comments in appgate/mod.rs and apps/fedimint/manifest.yml named a
specific node; the role is what carries the meaning, so the address is gone.
- CHANGELOG.md held five real addresses in published release notes — the most
exposed of the lot.
Deliberately NOT touched, because the plan's item-3 list is over-broad and
following it literally would break working code:
- 192.168.1.1 / .254, 192.168.0.0/16 and 100.64.0.0/10 are generic router
defaults, RFC1918 classification in backup_rpc, and CGNAT range logic in
pine_ha / CompanionIntroOverlay. Not leaked infra.
- `tx1138` is listed as a hostname to scrub but is two live things: the
user-facing default block explorer (`DEFAULT_TX_EXPLORER`) and
`RETIRED_TX1138_HOST`, the migration constant whose entire job is stripping
that retired registry from existing nodes' saved mirror lists. Scrubbing
either breaks a feature. The plan needs this correction.
- Android's `192.168.1.100` strings are UI placeholder text.
Item 5: added *.key, *.pem, id_rsa*, *.sqlite, *.db to .gitignore, with a
negation for core/archipelago/src/appgate/testdata/*.key. Checked those first —
they are documented throwaway TLS fixtures compiled in via include_bytes!, not
node identity — and the negation stops the new rule silently dropping them if
they are ever regenerated. Verified both directions: fixtures not ignored, a
stray key elsewhere caught.
Verified: residual grep for real infra addresses is clean; audit-secrets.sh
still 5/5; app-catalog drift 0 (the fedimint edit is a YAML comment, which does
not survive parsing into the signed catalog); 44/44 fips tests pass with the
rewritten assertion fixtures.
Note: these test runs shared the working tree with another agent's in-flight
LND work, which was present but unstaged and is not part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archi-dev-box was running archy-fedimint-ui and archy-lnd-ui with no
fedimint and no lnd container anywhere on the box. The Fedimint Guardian
UI sat on :8175 serving its "waiting for Bitcoin" page forever with
nothing behind it, which is what the operator reported as "fedimint
guardian installs but does not work" — there was nothing to install, the
UI was already up.
The boot reconciler drove companion provisioning from manifest_ids(),
which is every manifest the node can SEE: the whole apps/ directory plus
the signed-catalog overlay, 56 of them. The app reconciler has drawn this
line since phase 3 (ReconcileMode::ExistingOnly, "merely listing a
catalog manifest never installs an unqualified app"); the companion stage
never got the equivalent guard, so it stood up a UI for every app that
merely had a manifest and then self-healed it forever.
The other half is that reconcile() could only ever ADD. remove_for fires
only on the explicit uninstall RPC, so nothing ever subtracted: an
install that failed after its companion landed, or a container removed
by any other route, left a Restart=always unit alive permanently.
- installed_app_ids() replaces manifest_ids(): app ids whose container
actually exists. Returns Option, because a caller that removes things
on absence must not read "I could not look" as "nothing is installed".
Container presence in ANY state is the whole test — it deliberately
does not inherit the user_stopped/disabled filters, since a stopped app
is still an installed app and treating it otherwise would tear its
companion down and rebuild it on the next start.
- manifest_ids() is deleted rather than left unused. Its contract reads
as "installed" to anyone skimming, which is the whole bug.
- reap_orphans() removes companions whose backend is not installed, after
ORPHAN_GRACE (300s). The grace period is required, not defensive: this
node runs ARCHIPELAGO_USE_QUADLET_BACKENDS=true and a Quadlet app is
briefly containerless while restarting, so reaping on the first absent
tick would cost a healthy companion a teardown plus a possible 900s
image rebuild. A backend that reappears clears its clock.
- Reap failures are logged but kept out of the backoff input. Repair
keeps a companion available; reaping only tidies one away, and a wedged
reap must not back the repair path off to its 1h ceiling.
Every uncertain signal resolves toward not removing: no unit file and a
hung is-active reads as leave-it-alone.
Container suite 215/215.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backend verifies DID signatures as of f0c289a4, but the card only rendered
`trust_tier` / `trust_score`, so the verdict reached the frontend and died
there. Adds a badge next to the existing trust pill.
Deliberately a *separate* badge rather than folding into the trust tier: the
score blends relay count, provenance and policy compliance, while this answers
one narrow question — did the author prove control of the key their `author.did`
names. Merging them would hide the distinction that the signature layer exists
to draw.
- `valid` → green "signed" with a lock glyph
- `missing` → neutral grey "unsigned" (an unsigned publisher is unproven, not
hostile, so it reads as absence rather than alarm)
- `invalid` → red "bad signature". Discovery drops these before they reach the
cache, so it should be unreachable; rendered anyway so the UI
fails visibly rather than silently if that ever changes.
Two fail-safe details:
- The mapping defaults a missing field to `{status:'missing'}` rather than
leaving it undefined. A node on an older backend returns no field at all, and
"we couldn't check" must never render as "signed".
- The `invalid` arm is typed in the RPC client for the same reason: an
unhandled status falls through to "unsigned", not to the green badge.
The tooltip carries the meaning the two-word badge can't. "Signed" is easy to
misread as "safe", so it says what was actually proven — who published it — and
explicitly that this is not a statement about the app being safe.
Verified: vue-tsc clean, build green, and the new strings are present in the
freshly built Marketplace chunk (the build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/marketplace-protocol.md described a full authorship-verification chain and
was marked "shipped end-to-end". It wasn't: `signatures.manifest_hash` and
`signatures.did_signature` existed only as two struct fields that nothing read.
The authenticity actually delivered was the Nostr event's NIP-01 Schnorr
signature — which proves who *relayed* an event, not who *authored* the manifest
inside it. Anyone could republish someone else's manifest under their own DID.
Implemented:
- `canonical_signing_bytes` / `manifest_digest` — the signed preimage is the
manifest as canonical JSON (recursively sorted keys, no whitespace) with
`signatures` omitted, SHA-256'd. Canonicalisation is load-bearing, not
cosmetic: `container.env` is a HashMap with per-process random iteration
order, and `serde_json::Map` is only sorted while the `preserve_order` feature
stays off — a feature any crate in the graph can enable for everyone via
feature unification. Either would make the digest vary between runs, so
signatures would fail *intermittently*, which is far worse to diagnose than
failing cleanly.
- `sign_manifest` / `verify_manifest_signature` — Ed25519 over the 32 raw digest
bytes, verified against the key `author.did` encodes (reusing the existing
`identity::pubkey_bytes_from_did_key`).
- `publish` signs before broadcasting, fills `author.did` when empty, and
**refuses** to publish under a DID this node cannot sign as — otherwise we'd
spray manifests across every relay that every verifier then rejects.
- `discover` verifies before caching. A `missing` signature is a normal
unsigned publisher: listed, but earning no identity trust. An `invalid` one is
tampered or forged, so it is **dropped entirely** and logged — it fails closed
rather than appearing behind a warning badge a user can click through.
Trust scoring now requires proof for both identity-derived factors:
- The 30-point identity factor was `did.starts_with("did:")`. An unsigned
manifest with a plausible DID string and a pinned image scored 65 —
"Community" — on no cryptography at all. It now scores 35, "Unverified".
- **The 20-point federation factor is gated too**, which the original spec did
not say. An unverified `author.did` is just a string the publisher chose, so
without this an attacker could copy the DID of a peer the user federates with
and be rewarded for impersonating the party they trust most.
`marketplace.verify` now returns the signature verdict separately from the
advisory policy issues — `valid` has always meant "passes the advisory security
checks", so conflating it with authenticity would have been its own trap.
Tests (22 pass), weighted to the adversarial cases: tampering; tampering that
also rewrites `manifest_hash` while reusing the stolen signature; signing with
key A while claiming B's DID; undecodable did:keys including the old
`z6MkTest123` fixture that used to score 30/30; malformed base64 and
wrong-length signatures; digest stability across map insertion order; the digest
ignoring the `signatures` block; the federation-impersonation case; and a legacy
cache without the new field loading as `missing` rather than defaulting trusted.
Protocol doc rewritten so the preimage rules are normative — a third-party
implementation that canonicalises differently produces signatures we reject, so
"sorted keys, no whitespace, signatures omitted, sign the raw digest" now has to
be stated exactly rather than sketched.
Not included: surfacing the verdict in Marketplace.vue, which reads only
trust_score/trust_tier today. The field reaches the frontend; where the badge
goes is a UI call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`zbase32 0.1.2` is LGPL-3.0+ — the only hard copyleft dependency in the whole
Rust graph and the last remaining blocker for the MIT release
(docs/LICENSE-COMPLIANCE-AUDIT.md §2). Statically linking LGPL code into a Rust
binary obliges us to ship relinkable objects, which is impractical for a node
image.
The audit offered two routes: the MIT `z32` crate, or an original
implementation. Took the latter — z-base-32 is an alphabet substitution over a
bit stream, so ~60 lines removes the blocker while adding *zero* new
dependencies rather than trading one supply-chain entry for another.
**Byte-compatibility was the requirement, not a nice-to-have.** A `did:dht`
identifier IS this encoding of an Ed25519 public key, so any drift would
silently rotate every node's DID and orphan its already-published DHT records.
So the semantics were not guessed: I read the vendored zbase32-0.1.2 source to
extract exactly what `encode_full_bytes` and `decode_full_bytes_str` do —
including that decode truncates to the next lower byte boundary, which is why a
52-character string round-trips to 32 bytes while discarding 4 padding bits.
A model implementation was then validated against three independent sources
before any Rust was written, all five vectors agreeing:
encode(b"testdata", 64) -> qt1zg7drcf4gn (crate doctest)
encode_full_bytes("Just an…") -> jj4zg7bycfzn… (crate doctest)
decode_full_bytes("qb1ze3m1") -> b"peter" (crate doctest)
encode([f0,bf,c7]) -> 6n9hq (Zimmermann spec)
encode([d4,7a,04]) -> 4t7ye (Zimmermann spec)
The module pins all of those plus four known 32-byte keys, a 0..40-byte
round-trip sweep, a 52-char/round-trip check over 64 keys, rejection of the
characters z-base-32 deliberately omits (`l`, `v`, `2`, `0`) and of non-ASCII,
and an alphabet/decode-table consistency check so the compile-time reverse table
can't drift from the alphabet.
`did_dht.rs` gains `did_for_a_known_key_is_stable`, which pins the full
identifier string for a known key — the regression that would actually hurt,
asserted at the call site that gives the string its meaning.
Dropped from Cargo.toml and Cargo.lock (7 lines); no other user in the tree.
Verified: 28/28 network tests pass, zero copyleft crates remain in the lockfile.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The audit has claimed since 2026-07-23 that these were git-rm'd. They weren't —
only the web/dist copies went, and all of them were still tracked at HEAD nearly
three weeks later, in a repo about to be published under MIT.
Deleted (~40.7 MB):
neode-ui/public/assets/fonts/Courier_New/{CourierNew-Bold,CourierNew-Regular}.ttf
neode-ui/public/assets/fonts/Benton_Sans/BentonSans-Regular.otf
neode-ui/public/assets/fonts/Redacted/redacted.regular.ttf
neode-ui/public/packages/wireguard.apk (17 MB)
neode-ui/public/packages/atob.s9pk (23 MB)
Courier New is Monotype proprietary and Benton Sans is a commercial Font Bureau
typeface — neither is redistributable. wireguard.apk carries GPL-2.0 libwg
components, so shipping it triggers a source offer. atob.s9pk is a Start9
package of unknown license. Redacted's upstream is OFL-1.1 but no license text
was shipped; deleting was cheaper than sourcing it, since it was unused.
Verified unreferenced before deleting, not after:
- Every @font-face rule in the tree (2 in src/style.css, 2 in
public/entropy/index.html) loads Montserrat. None of these files was ever
loaded by CSS.
- The three `Courier New` hits (tailwind.config.js `mono`, two public HTML
font-family lists) name the *system* font as a fallback — they are not
@font-face sources, so rendering is unchanged.
- wireguard.apk and atob.s9pk have zero references in any tracked file.
- These live under neode-ui/public/, which Vite copies verbatim rather than
resolving, so their absence cannot break a build.
Deliberately kept: neode-ui/public/packages/archipelago-companion.apk, which IS
live (staged by .githooks/pre-push, the Android release flow, and the in-app
pairing QR); Montserrat (OFL.txt) and Open Sans (LICENSE.txt), both properly
licensed; and neode-ui/test-install.sh, which the same audit line listed but
which is not a licensing concern.
Audit updated: §1 and §3's font/package items marked closed, the false DONE
entry rewritten as a history note rather than deleted — a DONE line here is a
claim and should be re-verified with git ls-tree, which is exactly the lesson.
§2 (zbase32, LGPL-3.0+) is now the last hard blocker.
Side effect: ~40 MB off the frontend OTA tarball.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runbook still opened with "the catalog is accepted unsigned (migration
window) and the anchor is unpinned (RELEASE_ROOT_PUBKEY_HEX = None)". Both have
been true-for-a-while false: `trust::anchor::RELEASE_ROOT_PUBKEY_HEX` is a
`Some(...)` with a verification note in its doc comment, and
`releases/app-catalog.json` carries both a `signature` and a `signed_by`
did:key.
This one matters more than a normal stale status: a reader taking the header at
face value would think the fleet still accepts unsigned catalogs and that the
one-way anchor-pinning door is still open. It isn't — pinning already happened,
so any future ceremony is a *rotation*, which is the case the doc's own warning
about mismatched-signature hard-rejection applies to most sharply.
Marked complete and kept the procedure verbatim below, since it's exactly what a
key rotation or publisher change needs. Also dropped a stale `:21` line number
from the anchor.rs citation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inverse of the usual drift — these understate rather than overstate, which
is just as misleading for someone deciding what is safe to change.
**dht-distribution-design.md** was headed "Status: Design (no code yet)".
`core/archipelago/src/swarm/` has five modules plus `content_hash.rs`.
**phase4-streaming-ecash-plan.md** was headed "not implemented". `swarm/paid.rs`
states in its own header that it implements "DHT distribution plan, Phase 4 step
F", and there is a `streaming::` module behind five `streaming.*` RPCs
(list-services, configure-service, toggle-service, pay, prepare-payment).
Neither is reachable in a stock build, which is presumably why the headers were
never updated — and that is the part worth documenting rather than eliding. Both
now state the gates: the `iroh-swarm` cargo feature is off by default (iroh and
iroh-blobs are optional deps pulled in only by it), `config.swarm_enabled` is off
by default, and paid serving stays free for everyone until the operator enables
the `content-download` streaming service.
Checked the other plan-only docs for the same error; these two were the only
ones. `nostr-identity-import-plan.md`, `nostr-signer-login-research.md` and
`hardware-signer-design.md` correctly say no code exists — verified: no identity
import or NIP-07 login RPC, and no TROPIC01 reference anywhere in core.
`dual-ecash-design.md`'s "in progress" is right too — the `wallet.fedimint-*`
RPCs exist, no Cashu ones do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 2026-07-23 status block lists as DONE: "Deleted: Courier_New/, Benton_Sans/,
Redacted/ fonts; wireguard.apk; atob.s9pk; obsolete test-install.sh (all
git-rm'd)". All seven are still tracked at HEAD and present on disk. Only the
web/dist copies went; the sources never did.
git ls-tree -r HEAD --name-only | grep -iE 'Courier_New|Benton_Sans|Redacted/|wireguard.apk|atob.s9pk'
That means a repo about to be published under MIT still carries a commercial
Font Bureau typeface and two proprietary Monotype fonts — precisely what §3 of
this audit says must not ship. An audit that reports a blocker as closed is
worse than one that never checked, so the entry is now struck through with the
file list and the verification command inline.
Deleting them is safe and I checked before saying so: nothing references the
font *files* (the three `Courier New` hits are CSS font-family fallbacks naming
the system font, not @font-face sources), and wireguard.apk / atob.s9pk have
zero references anywhere in the tree. Left the deletion itself to the operator —
it is 40 MB of tracked binaries and outside a docs pass. Removing them also
takes 40 MB off the frontend OTA tarball, which is a separate open item.
Also re-verified the rest of the remaining list:
- `zbase32` (LGPL-3.0+) is still a direct dep (Cargo.toml:113, did_dht.rs:40,49).
Still the only hard copyleft blocker.
- LICENSE (MIT), NOTICE and both THIRD-PARTY-LICENSES inventories are present —
so the headline "no license of its own" is closed; softened the verdict to say
which blockers remain rather than leaving a stale "not releasable as-is".
- The four StartOS-derived crates still exist; flagged that KEY-05 cites
core/models, so that one needs review rather than a blind delete.
- Item 6 (git filter-repo history purge) is superseded — the launch plan is a
fresh-history publish, so there is no history to rewrite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**ADR-009** lists six "non-negotiable" mandatory security defaults. Checked each
against `core/container/src/manifest.rs` and `core/security/src/`:
- `seccomp_profile: Default` — the string `seccomp` appears **nowhere in
`core/`**. Not as code, not as a TODO. This constraint is entirely fictional.
- AppArmor — `container_policies.rs` generates and `apparmor_parser -r`s a
profile, but its own comment reads `TODO: Configure Podman to use the
profile`. `security.apparmor_profile` parses into a manifest field that
nothing ever reads.
- `user` UID > 1000 — no UID validation exists in the runtime parser at all.
- `image_tag` pinned — preflight script only; the parser accepts `:latest`.
- `readonly_root` / `no_new_privileges` — safe defaults when omitted, but
`validate_security()` never rejects an explicit `false`, so the ADR's
"Reject manifests that violate mandatory defaults" step does not exist.
Genuinely enforced: the capability allow-list and bind-mount confinement (the
latter stronger than the ADR describes). Added an Implementation status section
saying so per-row. The decision stands; the claim of enforcement did not, and on
a security ADR that gap is the whole point of writing it down.
**ADR-004** said Tor carries *all* inter-node communication and runs as the
`archy-tor` container. Neither holds: transport priority is mesh → LAN → FIPS →
Tor (`TransportKind` 1-4, Tor as last fallback, largely because of the latency
this ADR itself lists), and Tor is the host Debian service driven by
`archipelago-tor-helper` — `container-doctor.sh` actively removes an `archy-tor`
container if it finds one, and no `apps/tor` manifest exists. Added an amendment
rather than rewriting the record. Worth flagging that both changes landed
without their own ADR.
All 10 ADRs are Status: Accepted; 001-003, 005-008 and 011 verified consistent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Header claimed the whole 2026-04-22 plan "has been implemented". The
architecture was adopted, but checking each item against the tree:
- The `core/archipelago/src/reconcile/` module the doc lays out in detail —
desired.rs / current.rs / diff.rs / apply.rs / derived.rs / backoff.rs — was
never created. The reconciler shipped as container/boot_reconciler.rs +
container/prod_orchestrator.rs instead.
- FM2's named fix `reconcile::derived::render_bitcoin_conf` does not exist. The
drift was eliminated a different way: bitcoind runs with an explicit `-conf`
derived from secrets each start, and stale datadir configs are removed.
- FM1/FM3 are partial — companion UIs are Quadlet units, main app containers are
not, since `use_quadlet_backends` still defaults false. The "v1.7.48+ full
reconcile module / main containers become Quadlet units" step has not happened.
- **FM6 was never implemented.** There is no podman corrupt-state probe and no
`system renumber` recovery anywhere in the tree. The 2026-04 failure that made
a registry node unreachable would still require manual SSH today — which is
precisely the "zero-manual-intervention" target this doc opens with.
FM4 and FM5 did ship as described.
Replaced the blanket claim with a per-item table so the doc stays useful as
incident history without reading as a description of the code, and noted that
the unit path throughout says /etc/containers/systemd/ while units are actually
written to ~/.config/containers/systemd/ (the path is rootless).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_node_key_known_answer_vs_python_verifier` pinned the node Ed25519 and node
Nostr keys, and `test_release_root_known_answer` covers the release root. The
remaining six — FIPS mesh transport, identity Ed25519, identity Nostr (NIP-06),
Bitcoin BIP-84 and LND aezeed entropy — were only asserted to be mutually
distinct by `test_full_derivation_from_known_mnemonic`.
Distinctness is satisfied by ANY change to an HKDF info string or BIP-32 path.
So redefining `archipelago/lnd/entropy/v1` — the seed behind a user's Lightning
wallet — broke no test, while invalidating every backup verification a user had
already performed against docs/SEED-VERIFICATION.md. Same for the FIPS key that
authenticates a node on the mesh.
Expected values were produced independently by the Python verifier published in
that doc, whose primitives were themselves cross-checked against bip_utils and
cryptography's own HKDF (BIP-39 seed, both BIP-32 paths, x-only pubkey, bech32
and HKDF-SHA256 salt=None all matched byte for byte). This commit closes the
loop in the other direction: the Rust implementation now agrees with those same
bytes, so the doc and the code are pinned to each other.
Verified: 26/26 seed tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**container-lifecycle.md** told operators to read the reconciler's decisions with
`journalctl --user -u archipelago`. That returns nothing: `archipelago.service`
is a SYSTEM unit (`WantedBy=multi-user.target`) that merely runs as
`User=archipelago`. It's `sudo journalctl -u archipelago`. Easy to get wrong
because the companion Quadlet units next door genuinely are `--user`, so both
forms appear in the docs and only one is right per unit — spelled that out
inline. Swept the rest of docs/: no other instance.
**quadlet-compilation.md** — added the `Network=host` case. Podman rejects
`PublishPort` with host networking (crash-loop, exit 125), so the renderer drops
declared ports rather than emitting them
(`render_host_network_omits_publish_ports`). A developer reading the directive
list would otherwise expect a mapping that never appears.
Everything else in both docs verified against quadlet.rs / prod_orchestrator.rs /
boot_reconciler.rs: the unit dir, the DO-NOT-EDIT header, Pull=never,
DropCapability=ALL, Secret=…,type=env, TimeoutStartSec=0, RestartSec=10,
WantedBy=default.target, the render/write_if_changed/enable_now/disable_remove
four-step, uid 1000, adopt_existing, the user-stopped.json / user-uninstalled.json
desired-state gates, and the 30s tick.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reconciler's pre-start hooks for the btcpay stack, fedimint and fmcd
chowned unconditionally on EVERY prepare — and prepare re-runs far more
often than install (every reconcile that touches the app). archi-dev-box's
journal showed the same three dirs re-chowned every ~15s. The hooks exist
to repair old installs; they now skip when ownership is already correct
(root stat probe — the daemon's rootless metadata read can't see the
subuid-owned dirs).
Co-Authored-By: Claude <noreply@anthropic.com>
Ran the doc's script rather than only reading it, and cross-checked every
primitive it implements against independent libraries (bip_utils for BIP-39
seed / BIP-32 derivation / bech32, and cryptography's own HKDF): BIP-39 seed,
m/44'/1237'/0'/0/0, m/84'/0'/0', x-only pubkey, npub encoding and
HKDF-SHA256(salt=None) all match byte for byte. The hand-rolled crypto in this
doc is correct.
Two real gaps fixed:
- **The FIPS mesh transport key was missing.** `seed.rs:227` derives it from the
same master seed via `archipelago/fips/secp256k1/v1`, and a user verifying
their backup had no way to check it — despite it being the key that
authenticates them on the mesh. Added it to the diagram and as section 2b of
the script (same shape as the node Nostr key; verified against
`derive_fips_key` and `hkdf_derive` using `Hkdf::new(None, ikm)`).
- **The "compare with your node" commands were wrong.** The RPC endpoint is
`/rpc/v1`, not `/api/rpc`, and `identity.get-node` is not a method — the real
ones are `node.did` and `node.nostr-pubkey`. Also dropped "UI: Settings >
Identity", which is not a screen that exists, in favour of the two
identity files on disk.
Verified and left alone: all five other HKDF info strings, both BIP-32 paths,
and the `node_key.pub` filename. The release-root key
(`archipelago/release/root/ed25519/v1`) is deliberately still absent — it is
derived from the project's signing seed, not a user's node seed.
Noted separately: `system.get-node-key` sits in the CSRF-exempt list
(`api/rpc/mod.rs:337`) but has no dispatcher arm, so it is an exemption for a
method that does not exist. Harmless, but it should be removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This doc is marked "Status: implemented ... shipped end-to-end" and then
describes a cryptographic verification chain that does not exist. On a repo
about to go public, that is the single worst kind of doc bug: it promises a
security property.
`signatures.manifest_hash` / `signatures.did_signature` appear exactly once in
the codebase — as two struct fields at `marketplace.rs:106-107`. Nothing reads
them. There is no hash comparison, no DID resolution, no signature check. The
authenticity actually delivered is the Nostr event's own NIP-01 Schnorr
signature, which proves the publishing key sent the event but says nothing about
the DID the manifest names.
Added a warning at the top, marked the "Manifest Signing (DID Layer)" section
and steps 3-6 of the verification flow as not implemented, and annotated steps
7-8 as advisory (validate_manifest returns scoring issues; it does not block
discovery or install).
The trust model was overstated in the same direction:
- "DID Verification | 30 | Manifest is signed by a valid DID key" is a
`did.starts_with("did:")` string test. Any publisher can claim any DID and
take the 30 points.
- "Relay Consensus | 20" is graduated and never zero (1 relay still scores 5).
- "Version History | 15 | multiple published versions (shows maintenance)" —
nothing counts versions; it's 10 for a 3-part semver plus 5 for a non-empty
repo_url.
Worked the arithmetic through: an unsigned manifest with a plausible DID string
and a pinned image scores 65, landing in the "Community" tier. Said so.
Other corrections:
- `marketplace.unpublish` is documented but was never implemented (the string
appears nowhere); removed it and noted why NIP-33 makes it non-trivial. Added
the two payment methods that do exist (`create-invoice`, `check-payment`).
- The schema section said marketplace manifests "follow the existing
apps/{app-id}/manifest.yml schema", contradicting the header three paragraphs
above. They are separate types.
- The security-enforcement list claimed a capability allow-list, a
host-networking ban and system-path mount restrictions. Those rules are real
but live in the runtime manifest parser for a different schema — marketplace
validation checks four things and gates none of them.
- `run_as_user` documented as "> 1000" in two places while the code checks
`>= 1000` and the doc's own example uses 1000.
- Data-storage tree listed `cache/trust-scores.json` and `config.json`; neither
is ever written.
- The 15-minute cache TTL and 30-minute background refresh don't exist —
discovery is RPC-triggered and the cache has no expiry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The index covered users, architecture and app development but had no entry
point for "I want to work on Archipelago itself" — so eight tracked docs were
reachable only by guessing filenames, including the two that matter most to a
newcomer: developer-guide.md (how to build the workspace, the frontend and an
ISO) and LICENSE-COMPLIANCE-AUDIT.md (dependency licensing, which is exactly
what a reader checks first on an open-source repo).
Added a "Contributing to Archipelago itself" section covering those plus
bulletproof-containers, the signing runbook, the 1.8.0 hardening plan and
CLAUDE.md; filed pine-voice-commands under Getting started and demo-build-info
under contributing.
Also noted that ADR-010 was never issued — verified across all history, so the
009 → 011 gap is not a missing file — and added the two archived session logs
(HANDOVER-2026-07-02, SESSION-1.8.0-OTA-PROGRESS) to the archive table, which
already claimed to cover completed session logs but listed none.
Link check re-run across docs/: 0 broken. Only RELEASE_NOTES_BACKLOG.md is now
deliberately unindexed (internal working list).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
