Merge branch 'main' into worktree-reticulum-tcp-interop
This commit is contained in:
@@ -58,21 +58,32 @@ arbitrary app catalog to the entire fleet — fully unattended under
|
||||
`scripts/sign-manifest.sh` exists for re-signs. **Still open:** move the mirror
|
||||
to HTTPS + pinned cert (tracked with the next item); flip unsigned-manual-apply →
|
||||
hard-reject once the fleet is on a pinned-anchor binary.
|
||||
- [ ] 🔴 **Implement container image signature verification (cosign).**
|
||||
`container/src/podman_client.rs:255` — `pull_image(.., _signature)` silently discards
|
||||
the signature that the manifest threads all the way down
|
||||
(`prod_orchestrator.rs:1978/2435`). Wire `sigstore-rs`/`cosign verify` (or
|
||||
`podman pull --signature-policy`); hard-fail when a declared signature doesn't verify.
|
||||
- [x] 🔴 **Implement container image signature verification (cosign).** DONE 2026-07-04
|
||||
(code path; enforcement dormant until the ceremony): new `container::image_verify`
|
||||
gates BOTH pull sites (`PodmanClient::pull_image` + the dev-only `DockerRuntime`).
|
||||
Claims classify as None / the literal `cosign://...` placeholder (every fleet
|
||||
manifest today → pull proceeds, logged) / Declared → `cosign verify --key
|
||||
/etc/archipelago/cosign.pub --insecure-ignore-tlog=true` (+ both insecure-registry
|
||||
flags for the HTTP mirror; flags verified against cosign docs), hard-fail on missing
|
||||
key, missing cosign binary, timeout, or bad signature — a declared signature can
|
||||
never be skipped, on either runtime. Key path overridable via
|
||||
`ARCHIPELAGO_COSIGN_PUBKEY`. Deleted the caller-less, blocking, wrong-CLI
|
||||
`security::ImageVerifier`. **Activation = ceremony work**: pin cosign.pub on nodes +
|
||||
install cosign + publish real `image_signature` values (in that order); tracked with
|
||||
the Workstream B signing ceremony item.
|
||||
- [ ] 🟠 **Move the image mirror to HTTPS; drop `--tls-verify=false`.**
|
||||
`podman_client.rs:641` `INSECURE_REGISTRY_HOSTS = ["146.59.87.168:3000"]` +
|
||||
`config.rs:104,124` allowlist pull images over unauthenticated HTTP. Remove the raw-IP
|
||||
entries; give the mirror a valid/pinned cert. (Same host also baked insecurely into
|
||||
the ISO — see §F.)
|
||||
- [ ] 🟠 **Validate every image string at the pull site, not just the RPC boundary.**
|
||||
`is_valid_docker_image` runs in `install.rs:224`/`runtime.rs:549` but
|
||||
`prod_orchestrator::install_fresh` (1978) and `resolve_catalog_image` (944-971) pass
|
||||
catalog/manifest images straight to `pull_image`. Call the validator right before
|
||||
every pull.
|
||||
- [x] 🟠 **Validate every image string at the pull site, not just the RPC boundary.**
|
||||
DONE 2026-07-03: policy extracted to `container::image_policy` (single source of truth;
|
||||
RPC-boundary check delegates to it) and BOTH orchestrator pull sites (`install_fresh` +
|
||||
`ensure_resolved_source_available`) hard-bail on refs that fail it. Policy accepts
|
||||
trusted-registry refs + registry-less Docker Hub shorthand (`grafana/grafana` — used by
|
||||
8 manifests, can't name an attacker host); rejects any explicit non-allowlisted
|
||||
registry host, shell metachars, malformed refs. 4 new unit tests; container 159 /
|
||||
package 46 green.
|
||||
|
||||
---
|
||||
|
||||
@@ -83,11 +94,15 @@ atomic swap, single-depth backup). The gaps are **authenticity** (§A) and
|
||||
**verification depth** — plus the fact that the upgrade path has never run
|
||||
end-to-end on real hardware.
|
||||
|
||||
- [ ] 🔴 **Deepen the post-OTA health check.** `update.rs:456` (`probe_frontend_once`)
|
||||
passes on any 2xx/3xx from `GET /`, and `verify_pending_update` (494-593) only rolls
|
||||
back on that. A release with a broken RPC API, dead containers, or failed LND unlock
|
||||
passes and never rolls back. Add `/rpc/v1 update.status` + container-list/required-stack
|
||||
health assertions before clearing the pending-verify marker.
|
||||
- [x] 🔴 **Deepen the post-OTA health check.** DONE 2026-07-03: `verify_pending_update`
|
||||
now requires, in the same attempt, (1) frontend 2xx/3xx via nginx, (2) backend RPC
|
||||
liveness — unauthenticated POST `/rpc/v1`; 401/403 = alive, 5xx/404/refused = dead,
|
||||
so a 502-behind-static-files release now rolls back, (3) rootless `podman ps`
|
||||
reachability; plus a pre-loop binary-version==marker assertion that catches a silent
|
||||
or half swap (new frontend + old binary) deterministically. Per-app container
|
||||
assertions deliberately EXCLUDED — the pre-Quadlet service restart legitimately kills
|
||||
containers and the reconciler can need minutes (false-rollback risk); revisit after
|
||||
the Phase-3 flip. LND-unlock-level checks remain out of scope for the 90s window.
|
||||
- [ ] 🟠 **Run one real upgrade-from-vN-1 soak on hardware before tagging.**
|
||||
No test installs the previous version, points it at a staged 1.8.0 manifest, applies,
|
||||
and asserts health + rollback. This is the top release risk for an OTA release. A
|
||||
@@ -114,25 +129,60 @@ modules; production request/boot paths are essentially panic-free. The real risk
|
||||
sweep (`scheduler.rs`), block-header cache (`mesh/mod.rs`), 7× peer-transport badge
|
||||
(`sync.rs` + `content.rs`). Federation tombstone/untombstone upgraded to hard errors
|
||||
(see §I). Install-log line write left fire-and-forget with an explanatory comment.
|
||||
- [ ] 🟠 **Remove blocking `std::process::Command` from async handlers.**
|
||||
`install.rs:2222` `published_host_port` (sync podman on the install path),
|
||||
`dependencies.rs:316` (`df`), `system/handlers.rs:578` (`sudo`), `transport/fips.rs:50`
|
||||
(`systemctl`) stall tokio workers under load. Convert to `tokio::process` or
|
||||
`spawn_blocking`. Only 8 files use `std::process::Command` — bounded.
|
||||
- [ ] 🟡 **Restrict Bitcoin RPC exposure.** `bootstrap.rs:409` writes
|
||||
`rpcallowip=0.0.0.0/0`. Scope to the container subnet / `127.0.0.1`.
|
||||
- [ ] 🟡 **Move generated secrets from env to file mounts.** `manifest.rs:1208-1226`
|
||||
injects secrets as `-e KEY=value`, readable via `podman inspect` / `/proc/<pid>/environ`.
|
||||
Prefer bind-mounting the existing `0600` secret file or `podman --secret`.
|
||||
- [ ] 🟡 **Harden rate-limit IP extraction.** `middleware.rs:120-128` trusts
|
||||
client-spoofable `X-Real-IP`/`X-Forwarded-For` → per-request bucket rotation defeats the
|
||||
login limiter. Trust forwarded headers only from a configured proxy; have nginx set them.
|
||||
- [ ] 🟢 **Include `seq` in the mesh signed preimage.** `message_types.rs:245-288` signs
|
||||
`(t,v,ts)` but sets the anti-replay `seq` after signing → a radio MITM can alter ordering
|
||||
without breaking the signature.
|
||||
- [ ] 🟢 **Guard the short-DID slice panic** (`mesh/listener/decode.rs:566`) and gate the
|
||||
dev-mode `password123` bypass (`auth.rs:18`) behind `#[cfg]` before it can reach a
|
||||
release build.
|
||||
- [x] 🟠 **Remove blocking `std::process::Command` from async handlers.** DONE 2026-07-03:
|
||||
converted to `tokio::process` — `published_host_port` (install), `detect_disk_gb`
|
||||
(dependencies), factory-reset restart (system/handlers), `config.rs detect_host_ip`,
|
||||
the orchestrator host-facts helpers (`detect_host_ip/mdns/disk_gb`, `bitcoin_host`,
|
||||
`resolve_dynamic_env` now async through all 6 call sites), and `AutoRuntime::new`
|
||||
probes. `transport/fips.rs is_available()` (sync trait method on the async route path)
|
||||
now serves the cached value and refreshes via a background thread (stale-while-
|
||||
revalidate) instead of blocking on systemctl. `image_verifier.rs` cosign sites have no
|
||||
callers yet — handled with the §A cosign item. Tests: container 155 / transport 29 /
|
||||
config 29 / package 46 all green.
|
||||
- [ ] 🟡 **Restrict Bitcoin RPC exposure.** INVESTIGATED 2026-07-03 — the fix is NOT
|
||||
`rpcallowip`: under rootless-podman NAT every forwarded connection reaches bitcoind
|
||||
with an in-subnet source IP, so scoping `rpcallowip` blocks nothing (and container
|
||||
consumers use archy-net DNS anyway). The actual exposure is the host-side publish
|
||||
`8332:8332` (binds 0.0.0.0 → LAN can hit RPC, auth-only barrier; 4 write sites:
|
||||
`bootstrap.rs:424`, `package/config.rs:694`, `package/install.rs:1333/2450`, plus
|
||||
the knots manifest). Real fix = `127.0.0.1:8332:8332` host bind (P2P 8333 stays
|
||||
public; zmq 28332/28333 should get the same look — unauthenticated). ⚠ May break
|
||||
external wallets pointed straight at nodeIP:8332 — needs a user call + on-node gate
|
||||
re-run, so NOT changed drive-by from the dev box.
|
||||
- [x] 🟡 **Move secret env out of plaintext channels → podman secrets.** DONE 2026-07-05
|
||||
(code + unit tests; needs the on-node gate re-run before it counts as verified):
|
||||
secret env no longer merges into `environment` — it would land in `podman inspect`
|
||||
AND as plaintext `Environment=` lines in Quadlet unit files on disk (the worse leak).
|
||||
New pipeline: `expand_and_partition_env` taints plain entries that interpolate
|
||||
secrets (btcpay's `Password=${BTCPAY_DB_PASS}` connection strings travel as secrets
|
||||
too), values register as podman secrets (stdin, `--replace`, content-hash label,
|
||||
per-app cache so steady-state reconciles are podman-free), containers reference
|
||||
them via `secret_env` (API) / `Secret=…,type=env` (Quadlet). Verified empirically
|
||||
on fleet podman 5.4.2: value absent from inspect, runtime injection works. Rotation
|
||||
drift via `io.archipelago.secret-env-hash` container label; pre-upgrade containers
|
||||
lack the label → ONE-TIME recreate wave on first reconcile after deploy (by design —
|
||||
scrubs plaintext secrets from existing container configs). Docker dev fallback keeps
|
||||
plain env (no secret store). `/proc/<pid>/environ` inside the container is unchanged
|
||||
(env is the app-compat contract); the closed leaks are inspect output + unit files.
|
||||
- [x] 🟡 **Harden rate-limit IP extraction.** DONE 2026-07-03: the accept loop injects the
|
||||
TCP `PeerAddr` into request extensions; `extract_client_ip` honors
|
||||
`X-Real-IP`/`X-Forwarded-For` ONLY when the connection is from loopback (our nginx,
|
||||
which sets `X-Real-IP $remote_addr`) — direct connections (e.g. the FIPS peer
|
||||
listener) bucket under their socket IP, so per-request header rotation no longer
|
||||
defeats the login limiter. 3 unit tests.
|
||||
- [x] 🟢 **Include `seq` in the mesh signed preimage.** DONE 2026-07-04 (receiver half):
|
||||
`verify_signature` accepts a v2 preimage `(t,v,ts,seq)` alongside legacy v1 `(t,v,ts)`;
|
||||
`signed_with_seq()` is the v2 sender path, deliberately NOT yet wired — receivers
|
||||
hard-drop bad signatures, so senders stay on v1 until the whole fleet verifies v2.
|
||||
The seq-tampering window closes only when the v1 arm is removed (track as a
|
||||
post-fleet-rollout follow-up). Unit tests cover v2 verify, v2 seq-tamper rejection,
|
||||
and v1 sign-then-set-seq compatibility.
|
||||
- [x] 🟢 **Guard the short-DID slice panic** (`mesh/listener/decode.rs:566`) and gate the
|
||||
dev-mode `password123` bypass (`auth.rs:18`) behind `#[cfg]`. DONE 2026-07-04:
|
||||
advert_name uses `.get()` fallback (malformed radio-supplied DID can't panic the
|
||||
listener); the pre-setup dev-password login + the constant itself are
|
||||
`#[cfg(debug_assertions)]` — no release binary carries the bypass regardless of
|
||||
runtime config.
|
||||
- [ ] 🟢 **Apply the seccomp/apparmor profile** — `security/src/container_policies.rs:71` is a
|
||||
TODO; the profile is defined but never applied to podman.
|
||||
|
||||
@@ -159,9 +209,11 @@ The real issues are the app-bridge origin model and a bloated bundle.
|
||||
(precached by the service worker → blocks PWA install), plus ~18 MB of ~1 MB full-screen
|
||||
JPEGs. Convert backgrounds to WebP/AVIF at responsive sizes, lazy/stream the intro video,
|
||||
and exclude video/audio from the Workbox precache. Biggest, easiest perf win.
|
||||
- [ ] 🟢 **DOMPurify the `Server.vue` QR SVG** (`:283/:295` render `v-html` unsanitized while
|
||||
`TwoFactorSection.vue` sanitizes the analogous SVG); guard the unguarded `pollInterval`
|
||||
(`Mesh.vue:391`); surface silent data-fetch failures (`curatedApps.ts:58/71`).
|
||||
- [x] 🟢 **DOMPurify the `Server.vue` QR SVG / guard `Mesh.vue` pollInterval / surface
|
||||
`curatedApps.ts` fetch failures.** DONE 2026-07-03: WireGuard peer QR now sanitized with
|
||||
the same `USE_PROFILES: {svg}` call as TwoFactorSection; Mesh poll interval guarded +
|
||||
nulled on unmount; catalog fetch failures log per-URL console.warn incl. the
|
||||
all-sources-failed fallback. Bundle-verified.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -113,6 +113,32 @@ those are marked ✅ below with the commit that did it, so we stop re-litigating
|
||||
manifest-driven apps, never stacks; fedimint/fedimint-gateway/fedimint-clientd
|
||||
are 3 separate single-container apps with manifest dependency edges, not a
|
||||
coordinated stack. Workstream A's stack-migration tail is fully closed.
|
||||
- [ ] **Container thrashing/flapping + reconciler churn** (added 2026-07-04 — was
|
||||
implicit across other tracks, now an explicit pre-tag concern). The root cause
|
||||
of restart-storm flapping is pre-Quadlet architecture: restarting
|
||||
`archipelago.service` SIGKILLs every container in its cgroup, then the
|
||||
reconciler rebuilds the world over several minutes (the post-OTA health check
|
||||
deliberately skips per-app container assertions because of exactly this).
|
||||
Consolidated lever list, in order of impact:
|
||||
- **Phase-3 Quadlet default-flip** (tracked above) — removes the SIGKILL-the-world
|
||||
behavior entirely; the single biggest fix.
|
||||
- **Workstream F lifecycle items** — immich/grafana uninstall hangs + ghost
|
||||
containers, grafana reinstall stops, fedimint guardian sync
|
||||
(`docs/PRODUCTION-MASTER-PLAN.md` workstream F).
|
||||
- **Reconciler churn observability** — no metric/log today distinguishes "settling
|
||||
after restart" from "flapping"; add a per-app restart counter + log line when an
|
||||
app restarts >N times in M minutes so thrash is visible instead of anecdotal.
|
||||
- **Failed-unit self-healing gap (observed live 2026-07-06 on .228)**: fedimint's
|
||||
quadlet unit exited 255 at 21:21 and sat `failed` for 7+ hours — the reconciler
|
||||
never revived it (it repairs missing/drifted containers but doesn't
|
||||
`reset-failed`+start failed .services). Same for the indeedhub trio after the
|
||||
gate run. The health monitor also can't help (container is gone when the unit
|
||||
fails). Add a reconcile step: quadlet-backed app whose .service is `failed` and
|
||||
not user-stopped → reset-failed + start, with backoff.
|
||||
- Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01),
|
||||
indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`,
|
||||
removes executor stalls that made the API janky under reconcile load).
|
||||
- Perf polish riding along: 93 MB frontend dist shrink (hardening plan §D 🟡).
|
||||
- [ ] **Developer tooling CLI suite** (validate/render/local-install/lifecycle-test) —
|
||||
APP-PACKAGING-MIGRATION-PLAN.md step 5, needed before external devs can publish.
|
||||
- [x] ~~**Consolidated deploy 2026-07-01**: merged PR #67 (reticulum daemon
|
||||
|
||||
Reference in New Issue
Block a user