archy/.planning/phases/10-key-material-hardening/10-04-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
10-key-material-hardening	04	execute	2	10-03	scripts/security/host-secrets-audit.sh image-recipe/configs/archipelago-host-secrets-audit.service core/archipelago/src/bootstrap.rs core/archipelago/src/api/rpc/system/handlers.rs tests/first-boot-secrets/rotation-tests.sh docs/security/KEY-02-FLEET-ROTATION.md	false	KEY-02 KEY-04	truths artifacts key_links A deployed node can determine, from its own disk alone, whether its SSH host keys and TLS key are image-baked (fleet-shared) or per-node (D-06) The verdict is surfaced beyond a log file — it appears in system.stats so it is visible without shell access Rotation never loses remote access mid-flight: staging then atomic swap, reload rather than restart, and the new fingerprints recorded where an operator can read them Rotation does not happen by accident: the script is detect-only unless an explicit apply flag is passed, and the auto-versus-manual choice is a resolved human decision (D-06) Two real nodes flashed from the same ISO are proven to have distinct SSH host-key and TLS fingerprints (audit C-3) path provides contains min_lines scripts/security/host-secrets-audit.sh On-node detection of image-baked host secrets, and the guarded one-time rotation HOST_SECRETS_ROOT 100 path provides image-recipe/configs/archipelago-host-secrets-audit.service Boot-time detection unit, installed onto fleet nodes by the OTA runtime-asset promotion path provides contains core/archipelago/src/api/rpc/system/handlers.rs system.stats host_secrets field carrying the verdict and any rotation record host_secrets path provides min_lines tests/first-boot-secrets/rotation-tests.sh Automated harness for the detection verdicts and the guarded rotation 60 from to via pattern core/archipelago/src/bootstrap.rs image-recipe/configs/archipelago-host-secrets-audit.service run_runtime_assets installs the unit from the OTA runtime payload, the same path archipelago-doctor.service uses archipelago-host-secrets-audit from to via pattern scripts/security/host-secrets-audit.sh core/archipelago/src/api/rpc/system/handlers.rs script writes /var/lib/archipelago/host-secrets-audit.json, handler reads it into system.stats host-secrets-audit.json

Close the deployed half of F-03 (D-06): every node already in the field can determine whether it is running the fleet-shared SSH host key and TLS private key baked into its ISO, surface that verdict where an operator can see it, and rotate once — without losing remote access in the middle of the rotation.

Purpose: 10-03 stops the exposure growing. It does nothing for the nodes that are already live, which is exactly where the exposure sits. D-06 rejected builder-only for that reason.

Output: an on-node audit script delivered by the existing OTA runtime-asset promotion, a boot unit, a system.stats field, an automated harness, and recorded C-3 evidence from two real nodes.

D-06 is rated one-way and has its own decision checkpoint (Task 1) before the task that implements rotation. Rotating a host key invalidates every known_hosts entry for that node fleet-wide, including the Tailscale-reached nodes this project depends on for access. There is no going back to the old key.

<execution_context> @$HOME/.claude/gsd-core/workflows/execute-plan.md @$HOME/.claude/gsd-core/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/STATE.md @.planning/phases/10-key-material-hardening/10-CONTEXT.md @.planning/phases/10-key-material-hardening/10-03-SUMMARY.md @CLAUDE.md

<detection_method> How a node decides, from its own disk alone, whether its host secrets are fleet-shared.

No external data is available — the baked fingerprints live only in the ISO the node was flashed from, and older ISOs are not enumerable. Four on-disk signals, in priority order:

1. mtime against the first-boot anchor (primary, independent of any log). /var/lib/archipelago/.secrets-regenerated is touched at first boot on every node that ran the regeneration unit — including the fail-open path — so its mtime is a reliable "this node's first boot" anchor. Host keys regenerated at first boot carry an mtime within seconds of it; keys baked into the image carry the image build time, typically days or weeks earlier. A host key whose mtime is more than 300 seconds OLDER than the anchor came from the image and is therefore shared. Fall back to /root/.luks-archipelago.key (written by the installer with dd if=/dev/urandom, install-to-disk.sh) and then /etc/machine-id when the marker is absent; report unknown when no anchor exists rather than guessing.
2. The fail-open fingerprint (corroborating, and the audit's own C-3 criterion). .secrets-regenerated present AND /var/log/archipelago-first-boot-secrets.log containing a WARNING: line is precisely the combination the fail-open path at :1647/:1659/:1663 produces.
3. 10-03's durable failure record. /var/lib/archipelago/first-boot-secrets.failed present means a post-10-03 node failed regeneration and did not silently continue.
4. Rootfs provenance. Absence of /opt/archipelago/rootfs-identity-stripped means the node was flashed from a pre-10-03 ISO whose rootfs did carry baked material, so signal 1 is meaningful for it. Its presence means the rootfs shipped identity-free, so a missing host key is a fail-closed state rather than a shared one — a materially different verdict.

The verdict is one of per-node, shared, fail-closed-missing or unknown, and the evidence for it is always recorded alongside it. Never report per-node on the strength of an absent signal. </detection_method>

Task 1: Decide how host-key rotation reaches the fleet (D-06, one-way) docs/security/KEY-02-FLEET-ROTATION.md - .planning/phases/10-key-material-hardening/10-CONTEXT.md (decision D-06 and its one-way rating) - docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (finding F-03, and checklist item C-3 which is how the outcome gets verified) - core/archipelago/src/bootstrap.rs (lines 307-430 — run_runtime_assets, the OTA delivery path both options use) - CLAUDE.md (dev-pair-before-OTA policy, and the invariant that migrations never destroy data) Present the choice and stop. Do not implement either option before this resolves — Task 2's `` rating exists because walking through this door unattended is the failure mode the gate is for. When a fleet node detects that its SSH host keys and TLS key are the image-baked, fleet-shared ones, does it rotate them automatically, or detect and report and wait for an operator? Rotating an SSH host key is one-way: every existing `known_hosts` entry for that node breaks, on every machine that has ever connected to it, and there is no path back to the old key. The fleet is reached over Tailscale for day-to-day work, and several nodes are remote (`.228` is at a remote site and is in real use). A rotation that fires during an OTA on many nodes at once produces simultaneous host-key mismatches across the fleet with no warning. Against that: every boot a shared key stays in place is a boot on a key that anyone holding a copy of the published ISO also holds. D-06 already chose "remediate deployed nodes"; this decides only the trigger. Auto-rotate on the first boot after the OTA Closes the exposure on every node without operator effort; no node is left behind because someone forgot; the exposure window is bounded by the OTA rollout rather than by operator attention. Simultaneous fleet-wide `known_hosts` breakage with no advance notice; a node whose only access path is SSH-over-Tailscale becomes unreachable to any tooling that pins the host key until an operator clears the entry; if the rotation itself fails partway on a node, that node may be left needing physical console access. Detect and report on boot; rotate only when an operator runs the script with an explicit apply flag Access is never lost unexpectedly; the operator rotates one node at a time with the new fingerprint in hand; the verdict is still visible fleet-wide immediately via `system.stats`, so the exposure is measured rather than assumed; matches this project's standing "verify on the dev pair first" policy. The exposure persists on any node whose operator does not act; requires a follow-up operational task per affected node; a node that is never revisited stays exposed indefinitely. - The chosen option id is recorded verbatim in `docs/security/KEY-02-FLEET-ROTATION.md` under `## D-06 rotation trigger`, with the date and the reason given. - If `auto-on-boot` is chosen, Task 2 must additionally implement a pre-rotation reachability guard and a staged rollout knob, and the SUMMARY must record how a node is recovered if rotation fails mid-flight. - If `detect-report-then-apply` is chosen, Task 2's unit ships in detect-only mode and Task 3's checkpoint covers an operator-driven rotation on one node. Select: auto-on-boot, or detect-report-then-apply. The rotation trigger is a recorded human decision, not an implementation default. Task 2: On-node detection, guarded rotation, OTA delivery and status surfacing Rotating a node's SSH host key invalidates every existing `known_hosts` entry for it fleet-wide and cannot be undone — the old private key is destroyed by the swap (D-06). Task 1's decision is recorded in `docs/security/KEY-02-FLEET-ROTATION.md`; the script's default mode follows it. scripts/security/host-secrets-audit.sh, image-recipe/configs/archipelago-host-secrets-audit.service, core/archipelago/src/bootstrap.rs, core/archipelago/src/api/rpc/system/handlers.rs, tests/first-boot-secrets/rotation-tests.sh - core/archipelago/src/bootstrap.rs (lines 307-430 — run_runtime_assets: the `scripts` -> /opt/archipelago/scripts promotion, the chmod 755 sweep, and the `for unit in [...]` loop that installs units from image-recipe/configs) - image-recipe/configs/archipelago-doctor.service (the house pattern for a unit delivered this way) - image-recipe/_archived/build-auto-installer-iso.sh (10-03 output: the fail-closed script, its `FIRST_BOOT_SECRETS_ROOT` seam, the `first-boot-secrets.failed` record and the `rootfs-identity-stripped` provenance file this script keys off) - core/archipelago/src/api/rpc/system/handlers.rs (lines 179-206 — handle_system_stats, the object being extended) - core/archipelago/src/api/rpc/middleware.rs (line 41 — system.stats is in CACHEABLE_METHODS, so the new field must be cheap to compute) - tests/first-boot-secrets/run-tests.sh (10-03 output: the stub-PATH harness pattern to mirror) **a. `scripts/security/host-secrets-audit.sh`** (executable, `set -euo pipefail`). Mirrors 10-03's testability seam: `ROOT="${HOST_SECRETS_ROOT:-}"` prefixes every absolute path so the harness can drive it against a temp tree, and production behaviour with the variable unset is unchanged.

Modes: --detect (default, read-only) and --apply (rotates). --apply without --yes prints what it would do and exits 0 without touching anything, so a mistyped invocation is inert.

--detect evaluates the four signals from the detection_method section above, writes $ROOT/var/lib/archipelago/host-secrets-audit.json with fields verdict (per-node|shared|fail-closed-missing|unknown), evidence (an array of the signal strings that fired, each naming the file it read), checked_at (ISO-8601), ssh_host_key_fingerprints (the ssh-keygen -lf output for each public key — public data, safe to record) and tls_cert_sha256 (from openssl x509 -noout -fingerprint -sha256). Write it 0644 so the daemon can read it without privilege. Print a one-line human verdict to stdout. Exit 0 on any verdict — detection is informational and must never fail a boot.

--apply --yes rotates only the classes the detect pass flagged as shared, in this order, which is the access-preserving sequence and is the reason ordering is specified rather than left to the implementer:

1. Generate the replacement TLS keypair and the full SSH host-key set into staging directories. If any generation fails, abort before touching anything live and exit non-zero — a partial rotation is the failure mode that loses access.
2. Record the OLD fingerprints into $ROOT/var/lib/archipelago/host-key-rotation.json (0644: rotated_at, old_ssh_fingerprints, old_tls_sha256) BEFORE the swap, so an operator who loses access can still identify what changed.
3. Swap the TLS pair, then systemctl reload nginx.
4. Swap the SSH host keys, then systemctl reload ssh — reload, never restart. A reload re-execs the listener while already-forked session children keep running, so the operator's current SSH session survives its own rotation. Note that in the script comment; it is the single most important line in the file.
5. Append the NEW fingerprints to host-key-rotation.json, print them to stdout and to /dev/console (guarded so a missing console cannot fail the run), and re-run the detect pass so host-secrets-audit.json reflects the post-rotation state. Never delete a key without a successfully staged replacement in hand.

b. image-recipe/configs/archipelago-host-secrets-audit.service — Type=oneshot, After=archipelago-first-boot-secrets.service network.target, ExecStart pointing at /opt/archipelago/scripts/security/host-secrets-audit.sh with the mode Task 1's decision chose, WantedBy=multi-user.target. Follow image-recipe/configs/archipelago-doctor.service's shape.

c. core/archipelago/src/bootstrap.rs — add "archipelago-host-secrets-audit.service" to the for unit in [...] array at line ~361 so the OTA runtime-asset promotion installs it, exactly as archipelago-doctor.service is installed today. The scripts directory promotion at line ~320 already carries scripts/security/ along with it; confirm that by reading replace_dir_from_runtime rather than assuming, and record the confirmation in the SUMMARY. Enable the unit after install (systemctl enable --now via the existing host_sudo helper) in the same if changed block that already runs daemon-reload.

d. core/archipelago/src/api/rpc/system/handlers.rs — extend handle_system_stats's JSON with a host_secrets object read from /var/lib/archipelago/host-secrets-audit.json: verdict, checked_at, evidence, and rotated_at from host-key-rotation.json when present. A missing or unparseable file yields {"verdict":"unknown"} — never an error, because system.stats is cacheable and polled by the dashboard. Do not include the raw fingerprints in system.stats; expose them only in the on-disk record, so the polled dashboard payload stays small.

e. tests/first-boot-secrets/rotation-tests.sh (executable) — same stub-PATH pattern as 10-03's harness, driving host-secrets-audit.sh against temp roots:

• host keys newer than the anchor -> verdict per-node, JSON written, no files changed;
• host keys 30 days older than the anchor -> verdict shared;
• marker present plus a WARNING: line in the log -> verdict shared with both signals in evidence;
• rootfs-identity-stripped present and host keys absent -> verdict fail-closed-missing, not shared;
• no anchor at all -> verdict unknown;
• --apply without --yes -> no file in the tree changes (compare a find … -newer snapshot);
• --apply --yes on a shared tree -> old fingerprints recorded before the swap, new keys present, host-key-rotation.json contains both, and at no point in the run is the tree left with zero SSH host keys (assert by having the ssh-keygen stub fail and checking the live keys are untouched). bash tests/first-boot-secrets/rotation-tests.sh && cd core && CARGO_INCREMENTAL=0 cargo test -p archipelago system:: -- --nocapture <acceptance_criteria>
  • bash tests/first-boot-secrets/rotation-tests.sh exits 0 with a PASS line for each of the seven cases.
  • The abort-before-swap case is proven: with the ssh-keygen stub failing, the pre-existing host keys in the temp tree are byte-identical after the run (harness asserts with sha256sum).
  • cd core && CARGO_INCREMENTAL=0 cargo build -p archipelago succeeds and cargo clippy -p archipelago -- -D warnings is clean for the two touched Rust files.
  • A Rust test asserts handle_system_stats yields host_secrets.verdict == "unknown" when the JSON file is absent, and the recorded verdict when it is present.
  • grep -c 'archipelago-host-secrets-audit' core/archipelago/src/bootstrap.rs is at least 1.
  • grep -n 'systemctl reload ssh' scripts/security/host-secrets-audit.sh matches and there is no systemctl restart ssh in the file.
  • bash -n scripts/security/host-secrets-audit.sh exits 0; if shellcheck is available, shellcheck -S error is clean, otherwise its absence is recorded. </acceptance_criteria> A deployed node writes a verdict with its evidence to disk and to system.stats, and a guarded rotation exists that stages everything before touching anything live and reloads rather than restarts sshd.

Task 3: C-3 — two real nodes, distinct host keys, access intact Two nodes flashed from the same ISO are reachable, and both are running a build that carries this plan's runtime payload. docs/security/KEY-02-FLEET-ROTATION.md - scripts/security/host-secrets-audit.sh (Task 2 output) - docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (section 6, checklist item C-3 — the highest-value check in the audit's list) - docs/security/KEY-02-FLEET-ROTATION.md (Task 1's recorded decision, which determines whether step 4 below is expected to be a no-op) - CLAUDE.md (node access policy; `.228` is remote and in real use — do not rotate it uninvited) Claude prepares the sequence; the operator runs it on two nodes and pastes the output. Claude then records the result into `docs/security/KEY-02-FLEET-ROTATION.md` under `## C-3 — per-node host key and TLS uniqueness`, with node labels rather than addresses, and marks audit item C-3 as VERIFIED or FAILED with the fingerprints compared as opaque digests.

Pick the two nodes deliberately: use the dev pair (archi-dev-box + x250-dev) or another disposable pair. Do not run --apply against .228 or any node in real use as part of this checkpoint; if the audit verdict on such a node comes back shared, record it as a finding and raise it rather than rotating it inside a verification task. An on-node audit that reports whether this node's SSH host keys and TLS key are image-baked, and a guarded rotation that preserves the operator's own session. 1. On EACH node: sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect then cat /var/lib/archipelago/host-secrets-audit.json 2. On EACH node, capture the fingerprints directly, independently of the script: for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done and openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256 3. Compare the two nodes' outputs from step 2. ANY fingerprint appearing on both nodes is a confirmed F-03 instance — record it as the C-3 FAIL result, which is a finding, not an error. 4. If Task 1 chose detect-report-then-apply AND either node's verdict is shared: on ONE disposable node, from a session you are willing to lose, run sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes Then, WITHOUT closing that session, confirm it is still alive (echo still-here), open a SECOND connection and confirm the expected host-key-mismatch warning, accept the new key, and paste the new fingerprints from /var/lib/archipelago/host-key-rotation.json. 5. Confirm the web UI still loads over HTTPS on the rotated node (new self-signed cert, so a fresh browser trust prompt is expected and is the correct outcome). 6. Confirm the verdict propagated: call system.stats on the rotated node and paste the host_secrets object. <acceptance_criteria> - Step 3's comparison is recorded for both nodes: either "no fingerprint appears on both nodes" (C-3 PASS) or the exact shared fingerprint classes (C-3 FAIL, recorded as a live F-03 instance with the node labels). - If step 4 ran: the operator confirms the original session survived the rotation, the second connection showed the expected mismatch, and host-key-rotation.json contains both old and new fingerprints. - Step 6's host_secrets.verdict is per-node after a rotation, proving the detect pass re-ran and the surfacing works end to end. - docs/security/KEY-02-FLEET-ROTATION.md marks audit item C-3 VERIFIED (or FAILED-with-finding) with the date and node labels, and records every node whose verdict came back shared but which was deliberately NOT rotated, so none is quietly forgotten. </acceptance_criteria> Paste the per-node JSON, the fingerprint listings from both nodes, and (if run) the rotation output, then type "approved" — or describe what failed. Audit item C-3 is no longer UNVERIFIED: two real nodes are compared, any shared material is named, and a rotation has been demonstrated to preserve the operator's own session.

<threat_model>

Trust Boundaries

Boundary	Description
OTA runtime payload -> node filesystem	run_runtime_assets (bootstrap.rs:307-430) promotes scripts/ into /opt/archipelago/scripts and installs units into /etc/systemd/system using host_sudo. Anything shipped here runs as root at boot on every fleet node.
Rotation script -> live remote access	The script rewrites the credentials the operator's own SSH session and the web UI's TLS depend on.
system.stats -> dashboard	An authenticated, cacheable, frequently-polled read.

STRIDE Threat Register

Threat ID	Category	Component	Severity	Disposition	Mitigation Plan
T-10-31	Denial of service	Rotation loses remote access mid-flight, on a remote node, with no console	high	mitigate	Task 2 stages every replacement before touching anything live and aborts on any generation failure; swaps TLS then SSH; reloads rather than restarts sshd so forked session children survive; Task 3 step 4 proves it on a live session before it is trusted
T-10-32	Denial of service	Fleet-wide simultaneous known_hosts breakage during an OTA	high	mitigate	Task 1's blocking decision, with detect-report-then-apply available specifically to bound this; old and new fingerprints recorded before and after so operators can update known_hosts deliberately
T-10-33	Spoofing	A node keeps running a fleet-shared SSH host key that anyone with the published ISO holds	high	mitigate	Detection runs at boot and the verdict reaches system.stats, so an exposed node is visible without shell access; rotation closes it
T-10-34	Tampering	The audit script runs as root at boot from a directory replaced wholesale by the OTA payload	medium	mitigate	Delivery reuses the existing, already-trusted run_runtime_assets path and adds no new trust source; the script performs no network I/O and takes no input from the network; --apply requires --yes
T-10-35	Information disclosure	Host-key fingerprints and TLS digests written to disk and into a polled RPC payload	low	accept	Fingerprints of PUBLIC keys are public data; the private keys are never read by the script beyond regeneration. system.stats deliberately carries only the verdict, not the fingerprints
T-10-36	Repudiation	A rotation happens with no record of what the key used to be	medium	mitigate	Old fingerprints are written to host-key-rotation.json BEFORE the swap; new ones appended after; both echoed to console
T-10-37	Spoofing	A false per-node verdict from a missing signal leaves an exposed node looking clean	high	mitigate	The detection method reports unknown when no anchor exists and never infers per-node from an absent signal; every verdict carries the evidence strings that produced it
T-10-SC	Tampering	npm/pip/cargo installs	low	accept	No package-manager install occurs; the script uses openssl, ssh-keygen, stat and systemctl, all already present on fleet nodes, and the Rust change adds no crate. Executor MUST halt and raise a checkpoint if a new dependency appears necessary.
</threat_model>

<artifacts_this_phase_produces>

Artifacts this plan produces

New file: scripts/security/host-secrets-audit.sh (mode 755, promoted to /opt/archipelago/scripts/security/host-secrets-audit.sh by the OTA runtime payload)

Symbol	Kind	Contract
--detect	CLI flag	default; read-only; writes the verdict JSON; always exits 0
--apply	CLI flag	inert without --yes
--yes	CLI flag	required confirmation for a real rotation
HOST_SECRETS_ROOT	env var	path prefix for the whole script; unset in production
/var/lib/archipelago/host-secrets-audit.json	new on-disk file (0644)	verdict, evidence[], checked_at, ssh_host_key_fingerprints[], tls_cert_sha256
/var/lib/archipelago/host-key-rotation.json	new on-disk file (0644)	rotated_at, old_ssh_fingerprints[], old_tls_sha256, new_ssh_fingerprints[], new_tls_sha256

New file: image-recipe/configs/archipelago-host-secrets-audit.service — Type=oneshot, After=archipelago-first-boot-secrets.service network.target, WantedBy=multi-user.target.

Modified: core/archipelago/src/bootstrap.rs — archipelago-host-secrets-audit.service added to the runtime-asset unit install list.

Modified: core/archipelago/src/api/rpc/system/handlers.rs — system.stats gains a host_secrets object: { verdict, checked_at, evidence, rotated_at }. Absent file yields {"verdict":"unknown"}.

New file: tests/first-boot-secrets/rotation-tests.sh (mode 755) — seven cases.

New file: docs/security/KEY-02-FLEET-ROTATION.md — headings ## D-06 rotation trigger, ## C-3 — per-node host key and TLS uniqueness. </artifacts_this_phase_produces>

- `bash tests/first-boot-secrets/rotation-tests.sh` exits 0. - `cd core && CARGO_INCREMENTAL=0 cargo test -p archipelago` shows no new failures. - Both checkpoints resolved with pasted output. - Commit stages only this plan's six paths explicitly — never `git add -A`. Note that `core/archipelago/src/api/rpc/system/handlers.rs` and `core/archipelago/src/bootstrap.rs` are shared-tree files: run `git status --porcelain` first and, if another agent has uncommitted work in either, stop and raise it rather than committing around them.

<success_criteria>

• Every deployed node writes a verdict with its evidence and exposes it via system.stats.
• The rotation path stages everything before touching anything live, reloads rather than restarts sshd, and records old and new fingerprints on both sides of the swap.
• Rotation cannot happen by accident: detect-only default, --apply inert without --yes, and the auto-versus-manual trigger is a recorded human decision.
• Audit item C-3 is recorded as VERIFIED or FAILED-with-finding against two real nodes. </success_criteria>

Create `.planning/phases/10-key-material-hardening/10-04-SUMMARY.md` when done, carrying Task 1's chosen option, the seven harness results, the C-3 fingerprint comparison, the confirmation that `replace_dir_from_runtime` carries `scripts/security/`, and a list of any node whose verdict was `shared` but which was deliberately not rotated.