archy/.planning/phases/10-key-material-hardening/10-04-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	requirements-completed	coverage	duration	completed	status
10-key-material-hardening	04	fleet-host-secrets	security ssh-host-keys tls systemd ota rotation bash rust f-03	10-03: fail-closed first-boot secret generation, the /opt/archipelago/rootfs-identity-stripped provenance marker, and the /var/lib/archipelago/first-boot-secrets.failed durable failure record this script keys off	scripts/security/host-secrets-audit.sh — on-node detection of image-baked host secrets from the node's own disk alone, and a guarded one-time rotation Four-signal detection with a fixed precedence and per-verdict evidence strings, each naming the file it was read from Verdicts per-node | shared | fail-closed-missing | unknown — per-node is never inferred from an absent signal Per-key-class sharedness: SSH and TLS are judged and rotated independently, because a renamed node has a unique cert and shared host keys Access-preserving rotation: stage everything, abort before any swap, record old fingerprints first, TLS before SSH, mv-onto-path rather than rm-then-mv, reload sshd never restart archipelago-host-secrets-audit.service — detect-only boot unit delivered by the existing OTA runtime-asset promotion system.stats host_secrets object — the verdict visible without shell access tests/first-boot-secrets/rotation-tests.sh — 8-case harness through the HOST_SECRETS_ROOT seam, with four negative controls docs/security/KEY-02-FLEET-ROTATION.md — D-06's recorded decision, the C-3 result, and the not-yet-rotated register	ota-runtime-assets systemd system.stats sshd nginx-tls release-packaging	added patterns HOST_SECRETS_ROOT path-prefix seam, the same move FIRST_BOOT_SECRETS_ROOT makes in 10-03. Every property worth testing here is negative or ordering — 'touches nothing', 'aborts before any swap', 'records old fingerprints BEFORE the swap' — and none of them is assertable without the ability to force the failure. Ordering asserted by observation, not by content. The systemctl stub records whether the rotation record existed AT THE MOMENT of the first reload. Comparing fingerprints proves the right values were written; only this proves they were written first. Judge and remediate per key class, never per node. A node renamed via server.set-name has a freshly-minted TLS cert and untouched image-baked SSH host keys; a node-level verdict would call it clean. Precedence over accumulation: missing material can never be shared material, so the missing check runs first; direct evidence (the fail-open log line) outranks inference from timestamps.	created modified scripts/security/host-secrets-audit.sh image-recipe/configs/archipelago-host-secrets-audit.service tests/first-boot-secrets/rotation-tests.sh docs/security/KEY-02-FLEET-ROTATION.md core/archipelago/src/bootstrap.rs core/archipelago/src/api/rpc/system/handlers.rs scripts/create-release-manifest.sh	D-06 resolved as detect-report-then-apply. auto-on-boot would fire simultaneous fleet-wide known_hosts breakage during an OTA with no operator holding the new fingerprints, and — the argument that settled it — it cannot be dev-paired: by the time the behaviour has been observed on the dev pair it has already run everywhere, which contradicts the project's standing verify-on-the-dev-pair-first policy. The cost of the chosen option (exposure persists on any node nobody revisits) is bounded by visibility in system.stats and by a written register of nodes not yet rotated, not by automation. --apply writes NOTHING — not even its own verdict file — until --yes. 'Touches nothing' is worth being able to state without a footnote, and 'except for one file it rewrites' is what the footnote would have been. This also turns the dry-run test into an exact whole-tree comparison rather than one with a carve-out. --apply --yes refuses unless the verdict is `shared`. The guard against running it on the wrong node is structural rather than procedural — on a per-node node the command is inert even when typed deliberately and confirmed. Host keys are replaced by mv onto the existing path, not rm-then-mv. rm-then-mv opens a window in which the node has zero host keys on disk; sshd restarting into that window is unrecoverable remotely. Stale key types the new set does not include are removed only AFTER every staged key has landed — leaving an ssh_host_dsa_key behind would leave shared material behind. TLS is swapped before SSH. A dead web UI is recoverable over SSH; dead SSH on a remote node is not. Do the recoverable one first so a failure between the two leaves the recoverable path intact. reload sshd, never restart — stated in the script as the single most important line in the file. A reload re-execs the listener while already-forked session children keep running, so the operator survives their own rotation. The harness fails outright, before any case runs, if `systemctl restart ssh` ever appears in the file. A third sanctioned key producer is created, and said so loudly rather than quietly. Producer 1 (the ISO builder) is not present on a deployed node; producer 2 (TlsMaterial::regenerate) does TLS only and nothing in the daemon has ever rotated an SSH host key. The script header names all three and pins their shared parameters (rsa:2048, 3650 days, same subject and SAN, stage-parse-pair-check-swap) so they cannot drift apart. The verdict is never allowed to be optimistic. No anchor -> unknown; a standing first-boot-secrets.failed record -> unknown even when every mtime looks clean. T-10-37 is that a false per-node verdict leaves an exposed node looking clean, which is strictly worse than no verdict. system.stats carries the verdict and the evidence but NOT the fingerprints. They are public data, so this is not confidentiality — it is that a payload polled every few seconds should not carry digests an operator already on the node can read from disk. A unit test fails if a future edit forwards the whole file.		id description requirement verification human_judgment D1 A deployed node determines from its own disk alone whether its SSH host keys and TLS key are image-baked or per-node (D-06) KEY-02 kind ref status unit tests/first-boot-secrets/rotation-tests.sh cases 1-5 — per-node, shared-by-mtime, shared-by-fail-open-fingerprint, fail-closed-missing, unknown pass kind ref status other Real run on archi-dev-box: `sudo scripts/security/host-secrets-audit.sh --detect --json` -> per-node, anchored on /etc/machine-id, and its three fingerprints match an independent ssh-keyscan of the same host exactly pass false id description requirement verification human_judgment D2 The verdict is surfaced beyond a log file — it appears in system.stats so it is visible without shell access KEY-02 kind ref status unit handlers.rs::host_secrets_tests — 4/4: absent file -> unknown, unparseable file -> unknown, recorded verdict+evidence surfaced, rotated_at only when a rotation was recorded, fingerprints deliberately absent pass kind ref status other Never observed on a real node — needs a build carrying this plan deployed to the dev pair, then a system.stats call blocked true id description requirement verification human_judgment D3 Rotation never loses remote access mid-flight: staging then atomic swap, reload rather than restart, new fingerprints recorded where an operator can read them KEY-02 kind ref status unit rotation-tests.sh case 7a — old fingerprints on disk at the moment of the first reload (ordering observed, not inferred), keys replaced, `reload ssh` present and `restart ssh` absent from the systemctl log, verdict re-derived to per-node pass kind ref status unit rotation-tests.sh case 7b — with the SSH generator failing after TLS staging succeeded, the whole tree is byte-identical, no rotation record is written, and not one service is reloaded pass kind ref status other That a reload keeps the operator's own forked SSH session alive is proven by design, not by observation. UNVERIFIED on hardware. blocked true id description requirement verification human_judgment D4 Rotation does not happen by accident: detect-only default, --apply inert without --yes, and the trigger is a resolved human decision (D-06) KEY-02 kind ref status unit rotation-tests.sh case 6 — --apply without --yes exits 0 and not one byte of the tree changes, including the state dir; dry-run output warns that it is one-way pass kind ref status other docs/security/KEY-02-FLEET-ROTATION.md ## D-06 rotation trigger records `detect-report-then-apply` verbatim with the date and the reason; the shipped unit contains no apply path pass false id description requirement verification human_judgment D5 Two real nodes flashed from the same ISO are proven to have distinct SSH host-key and TLS fingerprints (audit C-3) KEY-02 kind ref status other docs/security/KEY-02-FLEET-ROTATION.md ## C-3 — **FAILED with finding**. Three distinct live nodes share all three SSH host keys; two also share their TLS private key. Gathered read-only via ssh-keyscan + anonymous TLS handshake; distinctness of the hosts confirmed via tailscale ping endpoints. fail kind ref status other Same-ISO provenance for those three nodes is UNVERIFIED — not required for the FAIL, but needed to bound how many other downloads carry the same keys blocked true id description requirement verification human_judgment D6 The OTA runtime-asset promotion actually delivers both the script and the unit to a fleet node KEY-02 kind ref status other replace_dir_from_runtime uses `cp -a src/. tmp` then `cp -a tmp/. dest` — recursive, so scripts/security/ rides along; the chmod sweep is `find dest -type f -name '*.sh' -exec chmod 755` with no -maxdepth, so the script lands executable. Read, not assumed. pass kind ref status other The unit had to be added to create-release-manifest.sh as well — bootstrap would have found nothing and installed nothing, silently. Neither half exercised end-to-end; needs a real release build and an OTA. blocked true	3h	2026-08-02	complete

Phase 10 Plan 04: Fleet host-secret detection and guarded rotation — Summary

Every deployed node can now say, from its own disk alone, whether it is running the SSH host keys and TLS private key baked into its ISO — the ones every downloader of that ISO also holds — and can be fixed once, by an operator, without losing remote access in the middle. Closes the deployed half of audit finding F-03.

C-3 FAILED, and that is the most important line in this document. Three live fleet nodes — archipelago-1, archy-x250-beta and archipelago — share all three SSH host key fingerprints. Two of them also share their TLS certificate, and therefore their TLS private key. F-03 is not theoretical on this fleet. None was rotated; all three are registered in docs/security/KEY-02-FLEET-ROTATION.md with the reason and the next step.

Commits

Commit	Task	What
96dba73a	1	D-06 recorded as detect-report-then-apply, with what the decision binds
0ed9334f	2	The audit script, the boot unit, the OTA wiring, the system.stats field, the 8-case harness
373c3bb3	2 (deviation)	Ship the unit in the OTA runtime payload — without this the whole plan was inert on arrival
a806a658	3	C-3 result: FAILED, with the finding, the method, and everything it does not establish

Nothing was pushed, tagged, built or deployed. No node was logged into, written to, or rotated.

Task 1 — D-06

detect-report-then-apply, recorded verbatim in docs/security/KEY-02-FLEET-ROTATION.md under ## D-06 rotation trigger.

The argument that settled it is not the one the plan anticipated. Both options were weighed on blast radius, but the decisive point is that auto-on-boot cannot be dev-paired. This project's standing policy is that nothing reaches the fleet before it is verified on archi-dev-box + x250-dev. A rotation that fires unattended on the first boot after an OTA has, by the time you have watched it happen on the dev pair, already happened everywhere. There is no observation point before the irreversible act.

Its cost is real and is written down rather than softened: any node whose operator does not act stays exposed indefinitely. It is bounded by making the verdict visible — system.stats, so an exposed node shows up without shell access — and by a written register of every node that reported shared and was not rotated. That register now has three entries in it, added by this plan's own verification.

Task 2 — detection, rotation, delivery, surfacing

How a node decides

Four signals in a fixed precedence, which matters more than the signals do:

1. Missing material can never be shared material. Checked first. On a 10-03-or-later node (/opt/archipelago/rootfs-identity-stripped present) an absent host key means generation never succeeded — fail-closed-missing, a materially different verdict, and rotation is not the remedy. Without the provenance marker the material is still absent, and the evidence says so rather than guessing.
2. The fail-open fingerprint outranks timestamps, because it is direct evidence rather than inference: .secrets-regenerated present plus a WARNING: line in /var/log/archipelago-first-boot-secrets.log. The two literal strings the pre-10-03 script emitted (WARNING: TLS regeneration failed, keeping baked key and WARNING: ssh-keygen -A failed, keeping baked host keys) also say which class survived, so the rotation narrows to it. An unrecognised WARNING: widens to both rather than guessing.
3. mtime against a first-boot anchor — .secrets-regenerated, falling back to /root/.luks-archipelago.key (written by the installer with dd if=/dev/urandom) and then /etc/machine-id. A key more than 300s older than the anchor carries the image build time.
4. The durable failure record (first-boot-secrets.failed) can only ever withhold a verdict, never grant one.

per-node is never inferred from an absent signal. No anchor → unknown. A standing failure record → unknown, even when every mtime looks clean. That is T-10-37: a false per-node leaves an exposed node looking clean, which is strictly worse than no verdict.

Every verdict carries the evidence strings that produced it, each naming the file it was read from, and the provenance signal is recorded on every run regardless of verdict because it changes what the other signals mean.

Judged per key class, not per node

This turned out to matter, and the C-3 scan is what proved it — see the archipelago finding below. SSH_SHARED and TLS_SHARED are tracked separately through detection and into rotation, so a node with a unique cert and shared host keys has only its host keys rotated.

The rotation, and why the order is specified

1. Stage everything first. Both the TLS pair and the full host-key set are generated into staging before anything live is touched, and any generation failure aborts with the tree untouched. A partial rotation is the failure mode that loses a node.
2. Record the OLD fingerprints before the swap. After the swap the old material is gone and unrecoverable; an operator who loses access anyway can still identify what changed.
3. TLS, then reload nginx. A dead web UI is recoverable over SSH. The converse is not. Do the recoverable one first.
4. SSH, then systemctl reload ssh — never restart. A reload re-execs the listener while already-forked session children keep running, so the operator survives their own rotation. Host keys are replaced by mv onto the existing path rather than rm then mv: the rm-then-mv shape opens a window in which the node has zero host keys on disk, and sshd restarting into that window is unrecoverable remotely. Stale key types the new set does not include are removed only after every staged key has landed.
5. New fingerprints to the record, to stdout and to /dev/console (guarded), then the detect pass re-runs so the verdict file reflects the post-rotation state.

Safety gates, in order of how likely each is to be the one that saves a node

• --detect is the default and is read-only.
• --apply without --yes writes nothing at all, not even its own verdict file.
• --apply --yes refuses unless the verdict is shared. On a per-node node the command is inert even when typed deliberately and confirmed. This is the guard against running it on the wrong node, and it is structural rather than procedural.
• The shipped unit contains no apply path at all, and says in a comment that adding one is a decision rather than a configuration change.
• ExecStart=- on the unit: a failed audit must never fail a boot.

A third key producer, declared

Producer 1 is gen_tls()/gen_ssh() in the ISO builder; producer 2 is TlsMaterial::regenerate in handlers.rs. Neither can do this job: producer 1 is not present on a deployed node, and producer 2 does TLS only — nothing in the daemon has ever rotated an SSH host key. So a third exists, and the script header names all three and pins what they must keep in common (rsa:2048, 3650 days, the same subject and SAN set, stage → parse both halves → prove they are a pair → swap) rather than leaving that to be rediscovered. tls_pair_matches() is carried over verbatim in intent from dad40c23.

Delivery — replace_dir_from_runtime confirmed by reading, not assumed

The plan asked for this to be confirmed rather than assumed. It was: replace_dir_from_runtime does cp -a "$src/." "$tmp" then cp -a "$tmp/." "$dest" — both recursive, so scripts/security/ rides along with the rest of scripts/. The executable sweep is find "$dest" -type f -name '*.sh' -exec chmod 755 {} + with no -maxdepth, so the script lands executable at /opt/archipelago/scripts/security/host-secrets-audit.sh.

The unit is added to the existing for unit in [...] array and enabled with --now, so the verdict lands with the OTA rather than at the next reboot.

Surfacing

handle_system_stats gains a host_secrets object read from the on-disk verdict. Three properties, because system.stats is in CACHEABLE_METHODS and the dashboard polls it: it never errors (absent, truncated or unparseable all yield {"verdict":"unknown"} — and every fleet node is in the absent case until the OTA lands, so that is the common path, not the edge one); it is two small file reads with no process spawn; and it carries no fingerprints. A unit test fails if a future edit forwards the whole file.

Harness — 8 cases, all green

host-secrets-audit.sh: 567 lines; bash -n clean
sshd handling: reload present, restart absent
PASS: host keys newer than the anchor -> per-node, JSON written, nothing else changed
PASS: host keys 30 days older than the anchor -> shared, evidence names both key classes
PASS: marker plus a WARNING: line -> shared, with both signals in evidence, despite per-node mtimes
PASS: identity-stripped rootfs with no host keys -> fail-closed-missing, not shared
PASS: no first-boot anchor -> unknown, never per-node
PASS: --apply without --yes -> exits 0 and not one byte of the tree changes
PASS: --apply --yes -> old fingerprints recorded BEFORE the swap, keys replaced, sshd reloaded not restarted, verdict re-derived
PASS: generation failure -> aborts before any swap; live keys byte-identical, no service reloaded

──────── host-secrets-audit summary ────────
passed: 8   failed: 0

Case 3 is dated so that the mtime signal alone would say per-node; if it passes it is because signal 2 fired. Case 7b forces the SSH generator to fail after TLS staging succeeded — the exact interleaving in which a naive implementation has already swapped the TLS pair.

Ordering is asserted by observation, not by content. Comparing the recorded old fingerprints against the pre-rotation keys proves the right values were written; it cannot prove they were written first. The systemctl stub therefore records, alongside each call, whether the rotation record existed at that moment. The first reload happens after the first swap, so rotjson=yes on that line is the ordering fact.

Negative controls — each reddens exactly one case

Each defect was reintroduced, the suite run, and the defect reverted.

Control A — the dry run writes its own verdict file ("one harmless file"):
FAIL: --apply without --yes -> STATE-DIR-CHANGED
passed: 7   failed: 1

Control B — old fingerprints recorded after the swap instead of before:
FAIL: --apply --yes -> OLD-FINGERPRINTS-NOT-RECORDED-BEFORE-THE-SWAP[reload nginx rotjson=no]
passed: 7   failed: 1

Control C — a failed SSH generation tolerated instead of aborting:
FAIL: generation failure -> exit-zero-on-aborted-rotation
      LIVE-MATERIAL-CHANGED-ON-AN-ABORTED-ROTATION
      rotation-record-written-for-a-rotation-that-never-happened
      no-loud-abort-on-stderr reloaded-a-service-during-an-aborted-rotation
passed: 7   failed: 1

Control D — per-node claimed with no anchor at all:
FAIL: no first-boot anchor -> verdict=per-node CLAIMED-PER-NODE-WITHOUT-EVIDENCE
passed: 7   failed: 1

Control B is the one worth noting: it reddens only because of the ordering observation. Every content-based assertion in case 7a still passes against that defect, because the right fingerprints do end up in the file — just too late to be of any use to someone who has lost access.

Control C also exposed a bug in my own harness (below).

Rust

running 4 tests
test ...host_secrets_tests::verdict_is_unknown_when_the_audit_file_is_absent ... ok
test ...host_secrets_tests::rotated_at_is_surfaced_only_when_a_rotation_was_recorded ... ok
test ...host_secrets_tests::verdict_is_unknown_when_the_audit_file_is_unparseable ... ok
test ...host_secrets_tests::recorded_verdict_and_evidence_are_surfaced ... ok
test result: ok. 4 passed; 0 failed

CARGO_INCREMENTAL=0 cargo build -p archipelago succeeds. cargo clippy -p archipelago produces zero diagnostics for bootstrap.rs and system/handlers.rs. Three warnings exist elsewhere in the crate (federation/handlers.rs unused import, mesh/flash.rs unused assignment, package/dependencies.rs dead const) — all pre-existing, all in other agents' files, none touched.

shellcheck is not installed on this machine, so shellcheck -S error was not run. Recorded rather than skipped silently. bash -n is clean on both new shell files.

Real run on this node

$ sudo scripts/security/host-secrets-audit.sh --detect --json
host-secrets: per-node — this node's SSH host keys and TLS key were generated here.
{
  "verdict": "per-node",
  "checked_at": "2026-08-02T18:57:08Z",
  "evidence": ["provenance: /opt/archipelago/rootfs-identity-stripped absent — this rootfs
    predates the 10-03 identity strip, so baked material is possible",
    "anchor: /etc/machine-id (machine-id, populated on this node's first boot), mtime
    2026-04-09T18:25:45Z",
    "per-node: every SSH host key and the TLS key is newer than the anchor, so all of it was
    generated on this node"],
  ...
}
$ ls -l /var/lib/archipelago/host-secrets-audit.json
-rw-r--r-- 1 root root 959 ...

archi-dev-box was installed from Debian directly, not flashed from the ISO, so it has no .secrets-regenerated marker and no first-boot log — it exercises the third fallback anchor. Its three fingerprints match an independent ssh-keyscan of the same host exactly, which is the only cheap cross-check available that the script's fingerprint extraction is correct against real tools.

--apply was never run outside a temp root, on this or any other machine.

Task 3 — C-3: FAILED, with finding

What was found

Three distinct live fleet nodes present byte-identical ECDSA, ED25519 and RSA host key fingerprints. Two of them also present the same TLS certificate, so they share the TLS private key.

Node	SSH host keys	TLS cert	Cert CN
archipelago-1	8WJplzKW… / lQgRXZ1n… / ym+gMOio…	62:F6:A6:02…	archipelago
archy-x250-beta	same three	same cert	archipelago
archipelago	same three	7C:6B:CD:98…	austin-sapien

archipelago-5, archi-dev-box, archy-dev-pa, framework-pt and shorty-s (.228) are each distinct from every other node and from each other. archy-x250-dev, archy-x250-pa, archy-x250-r2 and quantumterminal were unreachable and are UNVERIFIED.

Method, and why it is not the checklist's method

Gathered remotely and read-only: ssh-keyscan -T 6 <node> | ssh-keygen -lf -, and an anonymous TLS handshake for the certificate. No node was logged into, nothing was written, nothing was rotated.

This is a weaker instrument than C-3's on-node commands — it cannot read .secrets-regenerated, the first-boot log, or the ISO provenance. It was chosen because it needs no access and therefore covers the whole reachable fleet rather than two nodes, and because it is sufficient for the FAIL condition, which is any fingerprint appearing on two nodes.

Ruling out the boring explanation

Identical host keys are also what you would see if one machine were registered on the tailnet three times. Ruled out: all three answered a live TCP connection on port 22 within the same minute (one tailscaled serves one identity, so three simultaneously-live addresses are three hosts), they are owned by different tailnet accounts, and tailscale ping resolves them to different physical endpoints — archy-x250-beta answers from a different continent than the other two, which answer from the same NAT on different source ports.

The finding inside the finding

archipelago has a unique TLS cert and shared SSH host keys. Its cert CN is austin-sapien, not the image default — the signature of a node renamed through server.set-name, which re-mints the certificate via regenerate_tls_cert() so the SAN matches, and touches nothing else.

TLS uniqueness is therefore not evidence that a node's key material is per-node. Any renamed node gets a unique certificate for free while its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on certificates alone, this node would have looked clean. This is the concrete justification for judging and reporting the two key classes separately rather than issuing one node-level verdict — a design choice made before the scan, and vindicated by it.

Deliberately not rotated

All three are registered in docs/security/KEY-02-FLEET-ROTATION.md under "Nodes with a shared verdict, deliberately not rotated", with the reason and the next step. A checkpoint that remediates is a checkpoint that takes a node offline; archy-x250-beta in particular is reached over a DERP relay from another continent and is the least recoverable node in the set.

Deviations from Plan

1. [Rule 3 — Blocking] The unit could never have reached a node (373c3bb3)

• Found during: Task 2, tracing the delivery path end to end rather than trusting the plan's key_link.
• Issue: bootstrap.rs installs units from image-recipe/configs/ inside the OTA runtime payload, but scripts/create-release-manifest.sh copies only archipelago-doctor.service and .timer into that directory. archipelago-host-secrets-audit.service would never have existed on any node — src.exists() false, install skipped, no error and no log line. The entire deployed-node half of this plan would have shipped inert, and nothing would have said so.
• Fix: added the unit to that loop. The redundant if [ -f doctor.service ] || [ -f doctor.timer ] wrapper was removed at the same time — the per-unit -f test inside the loop already does that job, and the wrapper would have skipped the whole block on a tree carrying the new unit but not the doctor ones. A KEEP IN SYNC comment now names the array in bootstrap.rs, since two enumerations of one list in two languages in two files is what caused this.
• Scope: scripts/create-release-manifest.sh is outside this plan's files_modified. Taken because the alternative was to ship a deliverable that cannot reach its target and file the gap as a follow-up. Staged by path; git status --porcelain confirmed no other agent had uncommitted work in that file.

2. [Rule 1 — Bug in my own harness] set -o pipefail swallowed the summary

• Found during: negative control C.
• Issue: the failure-reporting path does diff <(…) <(…) | head -10. diff exits 1 when it finds differences, and under set -o pipefail that aborted the whole harness — so a case that failed by changing the tree killed the run before the summary line and before the remaining cases. Control A did not expose it, because its failure was a state-dir comparison rather than a tree diff, so diff exited 0.
• Why it matters more than it looks: the suppressed case is the one that detects a live rotation having modified files it should not have. A harness that dies silently on its most serious failure mode is worse than one that reports it noisily.
• Fix: || true on both reporting pipelines, with a comment naming the cause. Control C was re-run afterwards and the harness now exits 1 with the summary intact.

3. [Rule 2 — Correctness] --apply writes nothing at all, not just "nothing live"

• Issue: the natural implementation runs the detect pass and writes the verdict file before branching on mode, so --apply without --yes rewrites one file. Defensible, and it makes "touches nothing" a claim with a footnote.
• Fix: the write moved inside the --detect branch. --apply is now read-only in every path that does not reach a real rotation, and case 6 became an exact whole-tree comparison rather than one with a carve-out. Control A pins it.

4. [Rule 2 — Access preservation] mv onto the path instead of rm then mv

• Issue: the plan says "never delete a key without a successfully staged replacement in hand", which the ISO builder's gen_ssh satisfies with rm -f then mv. On a deployed node that still opens a window — small, but real — in which /etc/ssh holds zero host keys.
• Fix: each staged key is mv'd onto its live path (a rename(2), so atomic per key, and the directory is never empty), and only afterwards are key types the new set does not include removed — because leaving a stale ssh_host_dsa_key would leave shared material behind, which is the entire point of rotating.

Known Stubs

None. No placeholder values, no TODOs, no unwired code paths. Every path in the script is reached by at least one harness case.

Threat Flags

None. No new network endpoint, no new auth path, no schema change at a trust boundary. The script performs no network I/O and takes no input from the network; delivery reuses the existing, already-trusted run_runtime_assets path and adds no new trust source (T-10-34). No package was installed and no crate was added (T-10-SC: accept).

system.stats gains a field on an already-authenticated method (T-10-35: accept) and deliberately carries no fingerprints.

UNVERIFIED — exact evidence needed

Nothing below was observed. None of it is claimed as verified anywhere in this plan's output.

#	Item	Evidence needed
1	A rotation preserves the operator's own SSH session. The single most important behavioural claim in the plan, and it is proven by design only.	On ONE disposable node, from a session you are willing to lose: sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes; then, WITHOUT closing it, echo still-here; then a SECOND connection showing the expected host-key mismatch; then cat /var/lib/archipelago/host-key-rotation.json showing both old and new. The harness proves ordering and the abort path; it cannot prove systemctl reload ssh keeps a forked child alive.
2	host_secrets reaches system.stats on a real node.	A build carrying this plan deployed to the dev pair (archi-dev-box + x250-dev), then a system.stats call, then the same call after a rotation to confirm verdict flips to per-node and rotated_at appears. Proven against the file contract in unit tests only.
3	The OTA actually delivers script and unit.	A real scripts/create-release-manifest.sh run, then tar -tf the frontend tarball for archipelago-runtime/scripts/security/host-secrets-audit.sh and archipelago-runtime/image-recipe/configs/archipelago-host-secrets-audit.service; then on a node after the OTA: ls -l /opt/archipelago/scripts/security/host-secrets-audit.sh (expect mode 755) and systemctl status archipelago-host-secrets-audit.service.
4	The audit script's own verdict on the three shared-key nodes. Predicted shared; predicted is not observed.	sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect on archipelago-1, archy-x250-beta and archipelago, plus cat /var/lib/archipelago/host-secrets-audit.json. Needs the OTA, or the script hand-staged.
5	Same-ISO provenance for those three nodes. Not needed for the C-3 FAIL, but it bounds how many other downloads carry the same keys.	On-node: ls -l /opt/archipelago/rootfs-identity-stripped, cat /var/lib/archipelago/.secrets-regenerated, grep -i warning /var/log/archipelago-first-boot-secrets.log, plus whatever build id the installer recorded.
6	The four unreachable nodes (archy-x250-dev, archy-x250-pa, archy-x250-r2, quantumterminal).	Re-run the C-3 scan when they are online. archy-x250-dev is half the dev pair and has been offline 2 days.
7	The /dev/console leg of the rotation shout.	A real node or a VM console. Cannot be exercised in a temp root — the same limitation 10-03 recorded for its failure shout.
8	shellcheck -S error on both new shell files.	shellcheck is not installed on this machine. Install it and run it.
9	systemctl enable --now behaviour of the new unit.	A node. systemctl is-enabled archipelago-host-secrets-audit.service after an OTA.

Items 1, 2 and 4 are also recorded in .planning/WINDOWS.md (entries 11–13) so they remain visible at ship time.

Self-Check

• scripts/security/host-secrets-audit.sh — FOUND, mode 755, 567 lines, bash -n clean, contains HOST_SECRETS_ROOT
• image-recipe/configs/archipelago-host-secrets-audit.service — FOUND
• tests/first-boot-secrets/rotation-tests.sh — FOUND, mode 755, 458 lines, exits 0 with 8 PASS
• docs/security/KEY-02-FLEET-ROTATION.md — FOUND, contains ## D-06 rotation trigger and ## C-3 — per-node host key and TLS uniqueness
• grep -c 'archipelago-host-secrets-audit' core/archipelago/src/bootstrap.rs → 7 (≥1 required)
• grep -c 'host_secrets' core/archipelago/src/api/rpc/system/handlers.rs → 11
• grep -c 'host-secrets-audit.json' → 1 in handlers.rs (via HOST_SECRETS_AUDIT_FILE), 1 in the script — the key_link holds on both ends
• grep -n 'systemctl reload ssh' → line 415; grep -c 'systemctl restart ssh' → 0
• 10-03's harness re-run and still 9/9 green; image-recipe/_archived/build-auto-installer-iso.sh was not modified by this plan
• Commits 96dba73a, 0ed9334f, 373c3bb3, a806a658 — all FOUND
• git diff on the two shared Rust files inspected hunk by hunk before staging: additions only, all within host_secrets / the audit unit. No file belonging to plans 10-02, 10-06 or 01-18 (credentials/store.rs, device_tokens.rs, main.rs, seed.rs, session.rs, storage_crypto.rs, entropy.rs) was staged in any commit
• .planning/STATE.md and .planning/ROADMAP.md deliberately not updated — both carry other agents' uncommitted work in this shared tree, and the orchestrator owns them for this wave
• Nothing pushed, per the execution brief

Self-Check: PASSED