--- phase: 10-key-material-hardening plan: 04 type: execute wave: 2 depends_on: ["10-03"] files_modified: - scripts/security/host-secrets-audit.sh - image-recipe/configs/archipelago-host-secrets-audit.service - core/archipelago/src/bootstrap.rs - core/archipelago/src/api/rpc/system/handlers.rs - tests/first-boot-secrets/rotation-tests.sh - docs/security/KEY-02-FLEET-ROTATION.md autonomous: false requirements: [KEY-02, KEY-04] must_haves: truths: - "A deployed node can determine, from its own disk alone, whether its SSH host keys and TLS key are image-baked (fleet-shared) or per-node (D-06)" - "The verdict is surfaced beyond a log file — it appears in system.stats so it is visible without shell access" - "Rotation never loses remote access mid-flight: staging then atomic swap, reload rather than restart, and the new fingerprints recorded where an operator can read them" - "Rotation does not happen by accident: the script is detect-only unless an explicit apply flag is passed, and the auto-versus-manual choice is a resolved human decision (D-06)" - "Two real nodes flashed from the same ISO are proven to have distinct SSH host-key and TLS fingerprints (audit C-3)" artifacts: - path: "scripts/security/host-secrets-audit.sh" provides: "On-node detection of image-baked host secrets, and the guarded one-time rotation" contains: "HOST_SECRETS_ROOT" min_lines: 100 - path: "image-recipe/configs/archipelago-host-secrets-audit.service" provides: "Boot-time detection unit, installed onto fleet nodes by the OTA runtime-asset promotion" - path: "core/archipelago/src/api/rpc/system/handlers.rs" provides: "system.stats host_secrets field carrying the verdict and any rotation record" contains: "host_secrets" - path: "tests/first-boot-secrets/rotation-tests.sh" provides: "Automated harness for the detection verdicts and the guarded rotation" min_lines: 60 key_links: - from: "core/archipelago/src/bootstrap.rs" to: "image-recipe/configs/archipelago-host-secrets-audit.service" via: "run_runtime_assets installs the unit from the OTA runtime payload, the same path archipelago-doctor.service uses" pattern: "archipelago-host-secrets-audit" - from: "scripts/security/host-secrets-audit.sh" to: "core/archipelago/src/api/rpc/system/handlers.rs" via: "script writes /var/lib/archipelago/host-secrets-audit.json, handler reads it into system.stats" pattern: "host-secrets-audit.json" --- Close the deployed half of F-03 (D-06): every node already in the field can determine whether it is running the fleet-shared SSH host key and TLS private key baked into its ISO, surface that verdict where an operator can see it, and rotate once — without losing remote access in the middle of the rotation. Purpose: 10-03 stops the exposure growing. It does nothing for the nodes that are already live, which is exactly where the exposure sits. D-06 rejected builder-only for that reason. Output: an on-node audit script delivered by the existing OTA runtime-asset promotion, a boot unit, a `system.stats` field, an automated harness, and recorded C-3 evidence from two real nodes. **D-06 is rated one-way and has its own decision checkpoint (Task 1) before the task that implements rotation.** Rotating a host key invalidates every `known_hosts` entry for that node fleet-wide, including the Tailscale-reached nodes this project depends on for access. There is no going back to the old key. @$HOME/.claude/gsd-core/workflows/execute-plan.md @$HOME/.claude/gsd-core/templates/summary.md @.planning/PROJECT.md @.planning/STATE.md @.planning/phases/10-key-material-hardening/10-CONTEXT.md @.planning/phases/10-key-material-hardening/10-03-SUMMARY.md @CLAUDE.md **How a node decides, from its own disk alone, whether its host secrets are fleet-shared.** No external data is available — the baked fingerprints live only in the ISO the node was flashed from, and older ISOs are not enumerable. Four on-disk signals, in priority order: 1. **mtime against the first-boot anchor (primary, independent of any log).** `/var/lib/archipelago/.secrets-regenerated` is touched at first boot on every node that ran the regeneration unit — including the fail-open path — so its mtime is a reliable "this node's first boot" anchor. Host keys regenerated at first boot carry an mtime within seconds of it; keys baked into the image carry the image build time, typically days or weeks earlier. A host key whose mtime is more than 300 seconds OLDER than the anchor came from the image and is therefore shared. Fall back to `/root/.luks-archipelago.key` (written by the installer with `dd if=/dev/urandom`, `install-to-disk.sh`) and then `/etc/machine-id` when the marker is absent; report `unknown` when no anchor exists rather than guessing. 2. **The fail-open fingerprint (corroborating, and the audit's own C-3 criterion).** `.secrets-regenerated` present AND `/var/log/archipelago-first-boot-secrets.log` containing a `WARNING:` line is precisely the combination the fail-open path at `:1647`/`:1659`/`:1663` produces. 3. **10-03's durable failure record.** `/var/lib/archipelago/first-boot-secrets.failed` present means a post-10-03 node failed regeneration and did not silently continue. 4. **Rootfs provenance.** Absence of `/opt/archipelago/rootfs-identity-stripped` means the node was flashed from a pre-10-03 ISO whose rootfs did carry baked material, so signal 1 is meaningful for it. Its presence means the rootfs shipped identity-free, so a missing host key is a fail-closed state rather than a shared one — a materially different verdict. The verdict is one of `per-node`, `shared`, `fail-closed-missing` or `unknown`, and the evidence for it is always recorded alongside it. Never report `per-node` on the strength of an absent signal. Task 1: Decide how host-key rotation reaches the fleet (D-06, one-way) docs/security/KEY-02-FLEET-ROTATION.md - .planning/phases/10-key-material-hardening/10-CONTEXT.md (decision D-06 and its one-way rating) - docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (finding F-03, and checklist item C-3 which is how the outcome gets verified) - core/archipelago/src/bootstrap.rs (lines 307-430 — run_runtime_assets, the OTA delivery path both options use) - CLAUDE.md (dev-pair-before-OTA policy, and the invariant that migrations never destroy data) Present the choice and stop. Do not implement either option before this resolves — Task 2's `` rating exists because walking through this door unattended is the failure mode the gate is for. When a fleet node detects that its SSH host keys and TLS key are the image-baked, fleet-shared ones, does it rotate them automatically, or detect and report and wait for an operator? Rotating an SSH host key is one-way: every existing `known_hosts` entry for that node breaks, on every machine that has ever connected to it, and there is no path back to the old key. The fleet is reached over Tailscale for day-to-day work, and several nodes are remote (`.228` is at a remote site and is in real use). A rotation that fires during an OTA on many nodes at once produces simultaneous host-key mismatches across the fleet with no warning. Against that: every boot a shared key stays in place is a boot on a key that anyone holding a copy of the published ISO also holds. D-06 already chose "remediate deployed nodes"; this decides only the trigger. - The chosen option id is recorded verbatim in `docs/security/KEY-02-FLEET-ROTATION.md` under `## D-06 rotation trigger`, with the date and the reason given. - If `auto-on-boot` is chosen, Task 2 must additionally implement a pre-rotation reachability guard and a staged rollout knob, and the SUMMARY must record how a node is recovered if rotation fails mid-flight. - If `detect-report-then-apply` is chosen, Task 2's unit ships in detect-only mode and Task 3's checkpoint covers an operator-driven rotation on one node. Select: auto-on-boot, or detect-report-then-apply. The rotation trigger is a recorded human decision, not an implementation default. Task 2: On-node detection, guarded rotation, OTA delivery and status surfacing Rotating a node's SSH host key invalidates every existing `known_hosts` entry for it fleet-wide and cannot be undone — the old private key is destroyed by the swap (D-06). Task 1's decision is recorded in `docs/security/KEY-02-FLEET-ROTATION.md`; the script's default mode follows it. scripts/security/host-secrets-audit.sh, image-recipe/configs/archipelago-host-secrets-audit.service, core/archipelago/src/bootstrap.rs, core/archipelago/src/api/rpc/system/handlers.rs, tests/first-boot-secrets/rotation-tests.sh - core/archipelago/src/bootstrap.rs (lines 307-430 — run_runtime_assets: the `scripts` -> /opt/archipelago/scripts promotion, the chmod 755 sweep, and the `for unit in [...]` loop that installs units from image-recipe/configs) - image-recipe/configs/archipelago-doctor.service (the house pattern for a unit delivered this way) - image-recipe/_archived/build-auto-installer-iso.sh (10-03 output: the fail-closed script, its `FIRST_BOOT_SECRETS_ROOT` seam, the `first-boot-secrets.failed` record and the `rootfs-identity-stripped` provenance file this script keys off) - core/archipelago/src/api/rpc/system/handlers.rs (lines 179-206 — handle_system_stats, the object being extended) - core/archipelago/src/api/rpc/middleware.rs (line 41 — system.stats is in CACHEABLE_METHODS, so the new field must be cheap to compute) - tests/first-boot-secrets/run-tests.sh (10-03 output: the stub-PATH harness pattern to mirror) **a. `scripts/security/host-secrets-audit.sh`** (executable, `set -euo pipefail`). Mirrors 10-03's testability seam: `ROOT="${HOST_SECRETS_ROOT:-}"` prefixes every absolute path so the harness can drive it against a temp tree, and production behaviour with the variable unset is unchanged. Modes: `--detect` (default, read-only) and `--apply` (rotates). `--apply` without `--yes` prints what it would do and exits 0 without touching anything, so a mistyped invocation is inert. `--detect` evaluates the four signals from the detection_method section above, writes `$ROOT/var/lib/archipelago/host-secrets-audit.json` with fields `verdict` (`per-node`|`shared`|`fail-closed-missing`|`unknown`), `evidence` (an array of the signal strings that fired, each naming the file it read), `checked_at` (ISO-8601), `ssh_host_key_fingerprints` (the `ssh-keygen -lf` output for each public key — public data, safe to record) and `tls_cert_sha256` (from `openssl x509 -noout -fingerprint -sha256`). Write it 0644 so the daemon can read it without privilege. Print a one-line human verdict to stdout. Exit 0 on any verdict — detection is informational and must never fail a boot. `--apply --yes` rotates only the classes the detect pass flagged as `shared`, in this order, which is the access-preserving sequence and is the reason ordering is specified rather than left to the implementer: 1. Generate the replacement TLS keypair and the full SSH host-key set into staging directories. If any generation fails, abort before touching anything live and exit non-zero — a partial rotation is the failure mode that loses access. 2. Record the OLD fingerprints into `$ROOT/var/lib/archipelago/host-key-rotation.json` (0644: `rotated_at`, `old_ssh_fingerprints`, `old_tls_sha256`) BEFORE the swap, so an operator who loses access can still identify what changed. 3. Swap the TLS pair, then `systemctl reload nginx`. 4. Swap the SSH host keys, then `systemctl reload ssh` — reload, never restart. A reload re-execs the listener while already-forked session children keep running, so the operator's current SSH session survives its own rotation. Note that in the script comment; it is the single most important line in the file. 5. Append the NEW fingerprints to `host-key-rotation.json`, print them to stdout and to `/dev/console` (guarded so a missing console cannot fail the run), and re-run the detect pass so `host-secrets-audit.json` reflects the post-rotation state. Never delete a key without a successfully staged replacement in hand. **b. `image-recipe/configs/archipelago-host-secrets-audit.service`** — `Type=oneshot`, `After=archipelago-first-boot-secrets.service network.target`, `ExecStart` pointing at `/opt/archipelago/scripts/security/host-secrets-audit.sh` with the mode Task 1's decision chose, `WantedBy=multi-user.target`. Follow `image-recipe/configs/archipelago-doctor.service`'s shape. **c. `core/archipelago/src/bootstrap.rs`** — add `"archipelago-host-secrets-audit.service"` to the `for unit in [...]` array at line ~361 so the OTA runtime-asset promotion installs it, exactly as `archipelago-doctor.service` is installed today. The `scripts` directory promotion at line ~320 already carries `scripts/security/` along with it; confirm that by reading `replace_dir_from_runtime` rather than assuming, and record the confirmation in the SUMMARY. Enable the unit after install (`systemctl enable --now` via the existing `host_sudo` helper) in the same `if changed` block that already runs `daemon-reload`. **d. `core/archipelago/src/api/rpc/system/handlers.rs`** — extend `handle_system_stats`'s JSON with a `host_secrets` object read from `/var/lib/archipelago/host-secrets-audit.json`: `verdict`, `checked_at`, `evidence`, and `rotated_at` from `host-key-rotation.json` when present. A missing or unparseable file yields `{"verdict":"unknown"}` — never an error, because `system.stats` is cacheable and polled by the dashboard. Do not include the raw fingerprints in `system.stats`; expose them only in the on-disk record, so the polled dashboard payload stays small. **e. `tests/first-boot-secrets/rotation-tests.sh`** (executable) — same stub-PATH pattern as 10-03's harness, driving `host-secrets-audit.sh` against temp roots: - host keys newer than the anchor -> verdict `per-node`, JSON written, no files changed; - host keys 30 days older than the anchor -> verdict `shared`; - marker present plus a `WARNING:` line in the log -> verdict `shared` with both signals in `evidence`; - `rootfs-identity-stripped` present and host keys absent -> verdict `fail-closed-missing`, not `shared`; - no anchor at all -> verdict `unknown`; - `--apply` without `--yes` -> no file in the tree changes (compare a `find … -newer` snapshot); - `--apply --yes` on a `shared` tree -> old fingerprints recorded before the swap, new keys present, `host-key-rotation.json` contains both, and at no point in the run is the tree left with zero SSH host keys (assert by having the `ssh-keygen` stub fail and checking the live keys are untouched). bash tests/first-boot-secrets/rotation-tests.sh && cd core && CARGO_INCREMENTAL=0 cargo test -p archipelago system:: -- --nocapture - `bash tests/first-boot-secrets/rotation-tests.sh` exits 0 with a `PASS` line for each of the seven cases. - The abort-before-swap case is proven: with the `ssh-keygen` stub failing, the pre-existing host keys in the temp tree are byte-identical after the run (harness asserts with `sha256sum`). - `cd core && CARGO_INCREMENTAL=0 cargo build -p archipelago` succeeds and `cargo clippy -p archipelago -- -D warnings` is clean for the two touched Rust files. - A Rust test asserts `handle_system_stats` yields `host_secrets.verdict == "unknown"` when the JSON file is absent, and the recorded verdict when it is present. - `grep -c 'archipelago-host-secrets-audit' core/archipelago/src/bootstrap.rs` is at least 1. - `grep -n 'systemctl reload ssh' scripts/security/host-secrets-audit.sh` matches and there is no `systemctl restart ssh` in the file. - `bash -n scripts/security/host-secrets-audit.sh` exits 0; if `shellcheck` is available, `shellcheck -S error` is clean, otherwise its absence is recorded. A deployed node writes a verdict with its evidence to disk and to `system.stats`, and a guarded rotation exists that stages everything before touching anything live and reloads rather than restarts sshd. Task 3: C-3 — two real nodes, distinct host keys, access intact Two nodes flashed from the same ISO are reachable, and both are running a build that carries this plan's runtime payload. docs/security/KEY-02-FLEET-ROTATION.md - scripts/security/host-secrets-audit.sh (Task 2 output) - docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (section 6, checklist item C-3 — the highest-value check in the audit's list) - docs/security/KEY-02-FLEET-ROTATION.md (Task 1's recorded decision, which determines whether step 4 below is expected to be a no-op) - CLAUDE.md (node access policy; `.228` is remote and in real use — do not rotate it uninvited) Claude prepares the sequence; the operator runs it on two nodes and pastes the output. Claude then records the result into `docs/security/KEY-02-FLEET-ROTATION.md` under `## C-3 — per-node host key and TLS uniqueness`, with node labels rather than addresses, and marks audit item C-3 as VERIFIED or FAILED with the fingerprints compared as opaque digests. Pick the two nodes deliberately: use the dev pair (archi-dev-box + x250-dev) or another disposable pair. Do not run `--apply` against `.228` or any node in real use as part of this checkpoint; if the audit verdict on such a node comes back `shared`, record it as a finding and raise it rather than rotating it inside a verification task. An on-node audit that reports whether this node's SSH host keys and TLS key are image-baked, and a guarded rotation that preserves the operator's own session. 1. On EACH node: `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` then `cat /var/lib/archipelago/host-secrets-audit.json` 2. On EACH node, capture the fingerprints directly, independently of the script: `for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done` and `openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256` 3. Compare the two nodes' outputs from step 2. ANY fingerprint appearing on both nodes is a confirmed F-03 instance — record it as the C-3 FAIL result, which is a finding, not an error. 4. If Task 1 chose `detect-report-then-apply` AND either node's verdict is `shared`: on ONE disposable node, from a session you are willing to lose, run `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes` Then, WITHOUT closing that session, confirm it is still alive (`echo still-here`), open a SECOND connection and confirm the expected host-key-mismatch warning, accept the new key, and paste the new fingerprints from `/var/lib/archipelago/host-key-rotation.json`. 5. Confirm the web UI still loads over HTTPS on the rotated node (new self-signed cert, so a fresh browser trust prompt is expected and is the correct outcome). 6. Confirm the verdict propagated: call `system.stats` on the rotated node and paste the `host_secrets` object. - Step 3's comparison is recorded for both nodes: either "no fingerprint appears on both nodes" (C-3 PASS) or the exact shared fingerprint classes (C-3 FAIL, recorded as a live F-03 instance with the node labels). - If step 4 ran: the operator confirms the original session survived the rotation, the second connection showed the expected mismatch, and `host-key-rotation.json` contains both old and new fingerprints. - Step 6's `host_secrets.verdict` is `per-node` after a rotation, proving the detect pass re-ran and the surfacing works end to end. - `docs/security/KEY-02-FLEET-ROTATION.md` marks audit item C-3 VERIFIED (or FAILED-with-finding) with the date and node labels, and records every node whose verdict came back `shared` but which was deliberately NOT rotated, so none is quietly forgotten. Paste the per-node JSON, the fingerprint listings from both nodes, and (if run) the rotation output, then type "approved" — or describe what failed. Audit item C-3 is no longer UNVERIFIED: two real nodes are compared, any shared material is named, and a rotation has been demonstrated to preserve the operator's own session. ## Trust Boundaries | Boundary | Description | |----------|-------------| | OTA runtime payload -> node filesystem | `run_runtime_assets` (`bootstrap.rs:307-430`) promotes `scripts/` into `/opt/archipelago/scripts` and installs units into `/etc/systemd/system` using `host_sudo`. Anything shipped here runs as root at boot on every fleet node. | | Rotation script -> live remote access | The script rewrites the credentials the operator's own SSH session and the web UI's TLS depend on. | | `system.stats` -> dashboard | An authenticated, cacheable, frequently-polled read. | ## STRIDE Threat Register | Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | |-----------|----------|-----------|----------|-------------|-----------------| | T-10-31 | Denial of service | Rotation loses remote access mid-flight, on a remote node, with no console | high | mitigate | Task 2 stages every replacement before touching anything live and aborts on any generation failure; swaps TLS then SSH; reloads rather than restarts sshd so forked session children survive; Task 3 step 4 proves it on a live session before it is trusted | | T-10-32 | Denial of service | Fleet-wide simultaneous `known_hosts` breakage during an OTA | high | mitigate | Task 1's blocking decision, with `detect-report-then-apply` available specifically to bound this; old and new fingerprints recorded before and after so operators can update `known_hosts` deliberately | | T-10-33 | Spoofing | A node keeps running a fleet-shared SSH host key that anyone with the published ISO holds | high | mitigate | Detection runs at boot and the verdict reaches `system.stats`, so an exposed node is visible without shell access; rotation closes it | | T-10-34 | Tampering | The audit script runs as root at boot from a directory replaced wholesale by the OTA payload | medium | mitigate | Delivery reuses the existing, already-trusted `run_runtime_assets` path and adds no new trust source; the script performs no network I/O and takes no input from the network; `--apply` requires `--yes` | | T-10-35 | Information disclosure | Host-key fingerprints and TLS digests written to disk and into a polled RPC payload | low | accept | Fingerprints of PUBLIC keys are public data; the private keys are never read by the script beyond regeneration. `system.stats` deliberately carries only the verdict, not the fingerprints | | T-10-36 | Repudiation | A rotation happens with no record of what the key used to be | medium | mitigate | Old fingerprints are written to `host-key-rotation.json` BEFORE the swap; new ones appended after; both echoed to console | | T-10-37 | Spoofing | A false `per-node` verdict from a missing signal leaves an exposed node looking clean | high | mitigate | The detection method reports `unknown` when no anchor exists and never infers `per-node` from an absent signal; every verdict carries the evidence strings that produced it | | T-10-SC | Tampering | npm/pip/cargo installs | low | accept | No package-manager install occurs; the script uses `openssl`, `ssh-keygen`, `stat` and `systemctl`, all already present on fleet nodes, and the Rust change adds no crate. Executor MUST halt and raise a checkpoint if a new dependency appears necessary. | ## Artifacts this plan produces **New file:** `scripts/security/host-secrets-audit.sh` (mode 755, promoted to `/opt/archipelago/scripts/security/host-secrets-audit.sh` by the OTA runtime payload) | Symbol | Kind | Contract | |---|---|---| | `--detect` | CLI flag | default; read-only; writes the verdict JSON; always exits 0 | | `--apply` | CLI flag | inert without `--yes` | | `--yes` | CLI flag | required confirmation for a real rotation | | `HOST_SECRETS_ROOT` | env var | path prefix for the whole script; unset in production | | `/var/lib/archipelago/host-secrets-audit.json` | new on-disk file (0644) | `verdict`, `evidence[]`, `checked_at`, `ssh_host_key_fingerprints[]`, `tls_cert_sha256` | | `/var/lib/archipelago/host-key-rotation.json` | new on-disk file (0644) | `rotated_at`, `old_ssh_fingerprints[]`, `old_tls_sha256`, `new_ssh_fingerprints[]`, `new_tls_sha256` | **New file:** `image-recipe/configs/archipelago-host-secrets-audit.service` — `Type=oneshot`, `After=archipelago-first-boot-secrets.service network.target`, `WantedBy=multi-user.target`. **Modified:** `core/archipelago/src/bootstrap.rs` — `archipelago-host-secrets-audit.service` added to the runtime-asset unit install list. **Modified:** `core/archipelago/src/api/rpc/system/handlers.rs` — `system.stats` gains a `host_secrets` object: `{ verdict, checked_at, evidence, rotated_at }`. Absent file yields `{"verdict":"unknown"}`. **New file:** `tests/first-boot-secrets/rotation-tests.sh` (mode 755) — seven cases. **New file:** `docs/security/KEY-02-FLEET-ROTATION.md` — headings `## D-06 rotation trigger`, `## C-3 — per-node host key and TLS uniqueness`. - `bash tests/first-boot-secrets/rotation-tests.sh` exits 0. - `cd core && CARGO_INCREMENTAL=0 cargo test -p archipelago` shows no new failures. - Both checkpoints resolved with pasted output. - Commit stages only this plan's six paths explicitly — never `git add -A`. Note that `core/archipelago/src/api/rpc/system/handlers.rs` and `core/archipelago/src/bootstrap.rs` are shared-tree files: run `git status --porcelain` first and, if another agent has uncommitted work in either, stop and raise it rather than committing around them. - Every deployed node writes a verdict with its evidence and exposes it via `system.stats`. - The rotation path stages everything before touching anything live, reloads rather than restarts sshd, and records old and new fingerprints on both sides of the swap. - Rotation cannot happen by accident: detect-only default, `--apply` inert without `--yes`, and the auto-versus-manual trigger is a recorded human decision. - Audit item C-3 is recorded as VERIFIED or FAILED-with-finding against two real nodes. Create `.planning/phases/10-key-material-hardening/10-04-SUMMARY.md` when done, carrying Task 1's chosen option, the seven harness results, the C-3 fingerprint comparison, the confirmation that `replace_dir_from_runtime` carries `scripts/security/`, and a list of any node whose verdict was `shared` but which was deliberately not rotated.