--- phase: 10-key-material-hardening plan: 03 subsystem: iso-build tags: [security, iso, first-boot, systemd, ssh-host-keys, tls, machine-id, bash] requires: [] provides: - "Fail-closed first-boot per-device secret generation: the completion marker is written only when both TLS and SSH generation succeeded" - "A single producer per secret — gen_tls()/gen_ssh() are the only code in the ISO build that create the TLS keypair and SSH host keys" - "Retry-with-backoff inside a single boot, so a transient first-boot condition recovers without a reboot" - "archipelago-first-boot-secrets.timer — unattended self-heal every 15 minutes until generation succeeds" - "A build-time assertion that fails the ISO build if openssl or ssh-keygen is missing from the rootfs" - "Parse-back validation (openssl pkey / openssl x509) before the staging swap, so no service ever reads a truncated artefact" - "A durable failure record at /var/lib/archipelago/first-boot-secrets.failed plus console + journal + stderr on failure" - "FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF test seams on the generated first-boot script" - "An identity-free rootfs tar: no SSH host keys, no TLS keypair, machine-id truncated" - "/opt/archipelago/rootfs-identity-stripped build-time provenance marker" - "tests/first-boot-secrets/run-tests.sh — 6-case harness driving the shipped heredoc body against a temp root with stubbed generators" affects: [image-recipe, first-boot, sshd, nginx-tls] tech-stack: added: [] patterns: - "Seam-for-testability over assertion-in-a-comment: FIRST_BOOT_SECRETS_ROOT prefixes every absolute path so the NEGATIVE property (on failure the marker is NOT created) can be forced and asserted. Same move the entropy fix in 8b51b7e2 made for the RNG." - "Test the shipped bytes, not a copy: the harness extracts the first-boot script from the builder heredoc between the SECRETSSCRIPT delimiters, so the test and the artefact cannot drift." - "Strip identity material in the last Dockerfile layer so fail-closed is structural (no key exists) rather than procedural (a script promises to replace it)." key-files: created: - tests/first-boot-secrets/run-tests.sh - docs/security/KEY-02-ROOTFS-EVIDENCE.md modified: - image-recipe/_archived/build-auto-installer-iso.sh key-decisions: - "UNIFY, DO NOT DELETE. The defect in F-03 was never that a second attempt to create a key existed — it was that failure was silent and the marker lied about it. A second attempt is only dangerous when it is an unaudited second PRODUCER with its own idea of success, its own absent retry policy and its own absent failure record. So both secondary producers were folded out (the Dockerfile bake and the installer fallback) leaving one generator per secret, rather than 'keep the fallback' (leaves a silent second source) or 'delete the fallback and accept a dead node' (a false trade between security and UX)." - "Fail-closed governs SERVING; self-heal governs RECOVERING. These are separate properties and both must hold. Nothing serves on a key we did not generate; nothing dead-ends waiting for a human at a console." - "Self-heal uses a systemd timer, not a sleep loop in the script. A loop would hold a Type=oneshot open for hours and hide the failure from systemctl; the timer plus the service's existing ConditionPathExists=! costs a healthy node nothing and needs no teardown." - "The timer's enable uses a hand-written symlink fallback. Every other `chroot systemctl enable` here ends in `2>/dev/null || true`, which would silently drop the self-heal path — the one thing whose absence is invisible until a node is already broken." - "Consumers in `failed` state are explicitly restarted on success. try-reload-or-restart is a no-op on a failed unit, so without this a self-healed node would have valid keys on disk and nginx still down — recovery that isn't." - "Retry semantics: attempt count equals the number of FIRST_BOOT_SECRETS_BACKOFF entries, and the wait after the final attempt is skipped (a failed last attempt is terminal). With the default `2 8 20` that is 3 attempts at t=0s/2s/10s per generator; the trailing 20 is the ceiling if the list is lengthened. Documented in the script rather than left as a puzzle." - "`After=systemd-random-seed.service` added as its own unit line rather than appended to the existing After=local-fs.target, both because systemd accumulates After= lines and because the plan's acceptance criterion greps for exactly that string." - "/var/lib/dbus/machine-id is removed only when it is a real file, not when it is the symlink to /etc/machine-id that Debian normally ships. Deleting a live symlink risks a boot-time surprise for no gain; a real copy would be genuine shared state." requirements-completed: [] coverage: - id: D1 description: "A first-boot secret regeneration that fails does NOT set the completion marker, so the oneshot retries on the next boot (D-05)" requirement: KEY-02 verification: - kind: unit ref: "tests/first-boot-secrets/run-tests.sh#openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS" status: pass - kind: other ref: "Negative control: moving `touch \"$MARKER\"` back outside the success branch makes that case fail with MARKER-SET-ON-FAILURE (transcript below)" status: pass human_judgment: false - id: D2 description: "Each generator is retried with backoff within a single boot before the boot is declared failed (D-05)" requirement: KEY-02 verification: - kind: unit ref: "tests/first-boot-secrets/run-tests.sh#ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)" status: pass human_judgment: false - id: D3 description: "A terminal failure is loud: console + durable on-disk record + journal, not only a log file nobody reads (D-05)" requirement: KEY-02 verification: - kind: unit ref: "tests/first-boot-secrets/run-tests.sh#tls-fail case asserts first-boot-secrets.failed exists, names TLS, and stderr carries a FAILED line" status: pass - kind: other ref: "logger + tee -a $ROOT/dev/console emitted by shout(); the console leg cannot be exercised in a temp root and is UNVERIFIED on hardware" status: partial human_judgment: false - id: D4 description: "The shipped rootfs tar contains no SSH host keys, no TLS private key and no populated machine-id" requirement: KEY-04 verification: - kind: other ref: "docs/security/KEY-02-ROOTFS-EVIDENCE.md — requires an ISO build host; commands recorded, not yet run" status: blocked human_judgment: true - id: D5 description: "The regeneration script is exercised by an automated test that fails when the marker is set on a failed run" requirement: KEY-02 verification: - kind: unit ref: "tests/first-boot-secrets/run-tests.sh — 6/6 PASS; negative control reproduced" status: pass human_judgment: false - id: D6 description: "Exactly one producer per secret: no second code path anywhere in the ISO build can mint a TLS key or SSH host key with its own accounting" requirement: KEY-02 verification: - kind: unit ref: "tests/first-boot-secrets/run-tests.sh#single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh" status: pass - kind: other ref: "Negative control: reintroducing the installer's chroot openssl req block turns case 6 red naming the line, and nothing else" status: pass human_judgment: false - id: D7 description: "A failure self-heals unattended — it never dead-ends a node whose only exit is physical access" requirement: KEY-02 verification: - kind: unit ref: "tests/first-boot-secrets/run-tests.sh#self-heal: failed run then a later successful run -> key present, marker set, failed units restarted" status: pass - kind: other ref: "archipelago-first-boot-secrets.timer installed + enabled (symlink fallback) — UNVERIFIED on hardware; the harness proves the script half, not systemd's scheduling" status: partial human_judgment: false - id: D8 description: "The deterministic total-failure cause (missing generator binary) fails the BUILD, not the fleet" requirement: KEY-04 verification: - kind: other ref: "Rootfs RUN assertion on /usr/bin/openssl and /usr/bin/ssh-keygen; fires during the container build. UNVERIFIED until a build host runs it — see KEY-02-ROOTFS-EVIDENCE.md step 5b" status: blocked human_judgment: true duration: 1h completed: 2026-08-02 status: complete --- # Phase 10 Plan 03: Fail-closed first-boot secrets + identity-free rootfs — Summary First-boot per-device secret regeneration now retries with backoff and then fails closed, and the rootfs tar it repairs no longer carries the fleet-shared SSH host keys, TLS keypair or machine-id it was silently papering over. Closes the build side of audit finding **F-03**. > **Task 3 (C-4 build-host evidence) is a blocking checkpoint and is NOT done.** It needs an > ISO build host. `docs/security/KEY-02-ROOTFS-EVIDENCE.md` carries the exact command sequence > and is marked UNVERIFIED. Nothing in this plan claims the tar listing was observed. ## What was wrong `first-boot-secrets.sh` (a heredoc inside `image-recipe/_archived/build-auto-installer-iso.sh`, which is **live** — `image-recipe/build-debian-iso.sh` execs it) had two fail-open branches that logged `WARNING: ... keeping baked key` and continued, and `touch "$MARKER"` ran unconditionally **outside both `if` blocks**. Combined with the unit's `ConditionPathExists=!/var/lib/archipelago/.secrets-regenerated` and the script's own `[ -f "$MARKER" ] && exit 0`, one transient failure at first boot left that node on the image-wide shared SSH host key and TLS private key **permanently and silently** — and the ISO is a published artefact, so every downloader holds those keys. ## Commits | Commit | Task | What | |---|---|---| | `21043096` | 1 | Fail-closed, retried regeneration + `tests/first-boot-secrets/run-tests.sh` | | `408b328c` | 2 | Rootfs identity-strip layer + two comment corrections that follow from it | | `201ef474` | 3 (prep) | `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, marked UNVERIFIED | | `2efab5f2` | follow-up | Single-producer unification, build-time generator assertion, self-heal timer, 3 new test cases | | `d9b3a7d5` | follow-up | Quote the `Dockerfile.rootfs` heredoc so comments cannot execute (closes deferred D1) | | `40b77e39` | follow-up | Refuse to bless a cert minted under an untrustworthy clock; backdate `notBefore` | Nothing was pushed, tagged, built or deployed, per the execution brief. ## Task 1 — fail-closed regeneration - `ROOT="${FIRST_BOOT_SECRETS_ROOT:-}"` prefixes every absolute path. Unset in production the expansion is empty and behaviour is byte-identical; set, it is what makes the negative property assertable at all. - `retry()` runs each generator up to N times with waits from `FIRST_BOOT_SECRETS_BACKOFF` (default `2 8 20`). Staging-then-swap is preserved for both generators, with `.new` files removed on failure so no half-keypair is left behind. - `touch "$MARKER"` now lives inside a `TLS_OK == 1 && SSH_OK == 1` branch. Any other outcome writes `/var/lib/archipelago/first-boot-secrets.failed` (timestamp, which generator failed, both flags), shouts to console + `logger` + stderr, and `exit 1` so the unit lands in `failed` rather than `active`. A later successful boot deletes the record so a recovered node does not carry a stale alarm. - `After=systemd-random-seed.service` added to the unit. A no-op today (no seed file is baked, which the audit verified) and correct if one is ever introduced. - The script header states the operational trade in plain words, including that recovery from a terminal failure needs the physical console. ### Harness results (final, all six cases) ``` extracted 236 lines from the builder; bash -n clean PASS: both generators succeed -> exit 0, marker set, keys swapped in PASS: openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS PASS: ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts) PASS: TLS fails every attempt on a stripped root -> NO key, NO marker, non-zero exit, record names TLS PASS: self-heal: failed run then a later successful run -> key present, marker set, failed units restarted PASS: single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh ──────── first-boot-secrets summary ──────── passed: 6 failed: 0 ``` ### Negative control (required by the plan's acceptance criteria) `touch "$MARKER"` moved back outside the success branch, harness re-run, then reverted: ``` SCRATCH APPLIED: marker touch moved back outside the success branch --- harness against the fail-open variant --- extracted 175 lines from the builder; bash -n clean PASS: both generators succeed -> exit 0, marker set, keys swapped in FAIL: openssl fails every attempt -> MARKER-SET-ON-FAILURE exit=1 root=/tmp/tmp.Ta1YFhHWdi/root-tls-fail stderr: ARCHIPELAGO FIRST BOOT FAILED: could not generate this device's TLS key material. ... PASS: ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts) ──────── first-boot-secrets summary ──────── passed: 2 failed: 1 EXIT=1 ``` The test fails on exactly the regression it exists to pin, and only that case. ## Task 2 — identity-free rootfs Final `RUN` layer added to `Dockerfile.rootfs`, after every package install and after the `openssl req` layer, so nothing regenerates the material afterwards: - `rm -f /etc/ssh/ssh_host_*` (private keys and `.pub` alike) - `rm -f` the archipelago TLS key and crt, keeping the `/etc/archipelago/ssl` directory - `: > /etc/machine-id` (systemd's documented regenerate-on-next-boot state) - `/var/lib/dbus/machine-id` removed only if it is a real file, not the usual symlink - writes `/opt/archipelago/rootfs-identity-stripped` listing what it removed, with **no** timestamp so RECIPE_HASH reproducibility is unaffected The `openssl req` layer is deliberately unmodified. **RECIPE_HASH changed.** The strip layer is inside the hashed region (`sed -n '/^# STEP 1.../,/^# STEP 2.../p' | grep -c rootfs-identity-stripped` → 1), so the next build is forced to rebuild the rootfs tar. Task 3's evidence would be meaningless against a cached tar, and `--rebuild` is specified as well. `grep -c 'ssh_host'` on the builder went **3 → 6**. ## Deviations from Plan ### 1. [Rule 1 — Bug] Backticks in my own strip-layer comment would have hung every ISO build - **Found during:** Task 2, self-check of the added block. - **Issue:** `Dockerfile.rootfs` is written with an **unquoted** heredoc (`< TLS-KEY-EXISTS-AFTER-FAILURE TLS-CRT-EXISTS-AFTER-FAILURE passed: 5 failed: 1 ``` *(First run of this control also reddened case 5, because case 5's run-1 block redundantly re-asserted case 4's property. That assertion was removed — case 5 now tests recovery only — and the control re-run to confirm it is isolated. The transcript above is the re-run.)* **Control B — dead-end a node that has already failed once** (`exit 0` early if the failure record exists, a plausible "don't retry a known-bad node" optimisation): ``` FAIL: self-heal -> run2-marker-missing run2-key-missing run2-crt-missing run2-stale-failure-record run2-did-not-restart-failed-nginx passed: 5 failed: 1 ``` **Control C — reintroduce the installer's `chroot ... openssl req` block verbatim:** ``` FAIL: single-producer invariant -> SECOND-PRODUCER-at-line-3586 generator heredoc spans lines 1713-1950 of image-recipe/_archived/build-auto-installer-iso.sh passed: 5 failed: 1 ``` All three reverted; suite back to 6/6. ## Residual operational risk — stated plainly **A machine on which secret generation can never succeed ends up with no SSH host key and no TLS key. sshd will not start, nginx will not serve the web UI, and that node needs physical console access.** That is the honest worst case and it is not softened anywhere in the code comments either. What shrinks it to genuinely-broken-hardware: - **The deterministic cause is gone before shipping.** A missing `openssl` or `ssh-keygen` fails the ISO build, so it cannot reach a node. - **Transient causes are absorbed.** Three attempts with backoff inside the boot (proven by harness case 3, which shows a generator failing twice and succeeding on the third), then every 15 minutes on the timer, then again on every boot — indefinitely, because the marker is never written on failure. - **Recovery completes itself.** On a later success the script restarts the units that refused to start, so the node comes back without a reboot and without a human (harness case 5). What is left is a machine where `openssl` or `ssh-keygen` is present but cannot ever produce a key — a disk that is permanently full, or failing hardware. On that machine the node refuses to serve rather than serving on a key nobody generated, which is the trade this phase exists to make. It says so on the console, in the journal, and in `/var/lib/archipelago/first-boot-secrets.failed`. ## Follow-up: quote the Dockerfile heredoc (`d9b3a7d5`) — closes deferred D1 `cat > "$WORK_DIR/Dockerfile.rootfs" < # the archipelago backend calls `systemctl start archipelago-fips.service` < # fips-gateway is gated behind the Cargo feature (depends on > # fips-gateway is gated behind the `gateway` Cargo feature (depends on ``` **Case 7** asserts every heredoc writing `Dockerfile.rootfs` has a quoted delimiter, and when one does not, reports which body lines would execute. The assertion is on the **delimiter, not on backticks** — with quoting a backticked comment is legal, and six of them are back in the body on purpose; flagging backticks would flag a non-bug and fail on the very comments this restored. Controls: ``` Control D — unquote the delimiter (the real regression): FAIL: Dockerfile heredoc quoting -> UNQUOTED-DELIMITER-at-line-287 would-execute-at-lines:317,318 passed: 6 failed: 1 Control E — add a backticked comment, delimiter still quoted: PASS: Dockerfile heredoc delimiters are quoted — a backticked comment cannot execute passed: 7 failed: 0 and it renders intact: 175:# Control E: a backticked `systemctl start archipelago-fips.service` comment ``` Control E is the more informative of the two: the backtick that used to be a build-host RCE is now inert and renders as written. That is what "fixed the class" means, and it is why a bare backtick reintroduction correctly reddens nothing. `deferred-items.md` held D1 as its only entry and has been **deleted** — nothing was left that is genuinely out of scope. ## Follow-up: untrustworthy clock at cert-minting time (`40b77e39`) The failure fail-closed cannot catch, **because generation succeeds**. This unit runs before time has synced; `openssl req -x509` stamps `notBefore` from whatever the clock says. Dead RTC or flat CMOS battery → clock ahead gives "not yet valid" (harder to diagnose than a self-signed warning), clock behind gives an already-expired cert once time syncs. The marker was then set and never revisited: a node permanently serving a cert nothing accepts. **Finding, established rather than assumed:** this image does **not** use `systemd-timesyncd`. It installs and enables **chrony** (`:388`, `:575`), and `chrony-wait.service` — the unit that is `Before=time-sync.target` — is **not** enabled. So `time-sync.target` is inert here and ordering `After=` it would buy nothing. Enabling `chrony-wait` to make it meaningful would stall boot behind NTP on a node with no network, and these nodes are routinely offline at first boot. **Decision: no ordering change.** Not deadlocking boot outranks cert-date elegance (constraint 3). Fixed locally instead: 1. **Backdate `notBefore` by 24h** so ordinary node/client skew cannot invalidate a fresh cert. This does not weaken a self-signed cert — `notBefore` is not a security control here. `-not_before`/`-not_after` arrived in OpenSSL 3.5 and the rootfs is `debian:trixie` which ships it, but the capability is **probed, not assumed**: guessing wrong would fail every attempt and brick the node, the exact outcome all of this exists to prevent. Without the flags we do not backdate, and rule 2 still covers the dangerous case. 2. **Refuse to bless a cert dated outside a plausible window** (2026-01-01 … 2056-01-01). The material stays installed — the node is usable, sshd comes up — but the dates are recorded as `failed=cert-dates` and the cert is regenerated automatically once time syncs. Generation is now driven by **need** rather than "is the marker absent", and `ConditionPathExists=!` was removed from the unit so a node that already completed can still be re-examined. Skipping the unit is precisely how such a node would stay broken forever. On a healthy node the script exits in milliseconds. **Anti-spin is one condition:** a date-driven regeneration happens *only* when the clock is currently plausible. A node whose clock is still wrong re-checks and mints nothing. ### Regression I introduced and caught Driving generation purely by content made `needs_ssh()` false whenever *any* host key existed — which would have left an image-baked, fleet-shared host key in place forever. **That is F-03 reopened.** The marker check is back in both `needs_` functions. Case 1 — which prestages a baked key and asserts it was replaced — is what caught it. ### Controls ``` Control F — bless the cert regardless of clock (the pre-fix behaviour): FAIL: wrong clock -> run1-BAD-DATES-NOT-RECORDED passed: 7 failed: 1 Control G — remove the anti-spin guard: FAIL: wrong clock -> SPINNING-reminted-while-clock-still-wrong(1->2) passed: 7 failed: 1 ``` **Control G first passed against a deliberately broken guard**, which was a flaw in my test, not in the fix: the assertion compared certificate dates, and a re-mint under a frozen fake clock produces a byte-identical `notBefore`. Dates cannot distinguish "left alone" from "regenerated again". The assertion now counts `openssl req` invocations, which can — and only then did the control redden. Worth recording as the second time in this plan that a first-draft assertion looked green for the wrong reason. ### Not covered here Nodes already deployed from earlier ISOs **never receive this script** — it is installed by the installer, not shipped by OTA. Fleet remediation for those nodes is 10-04/OTA work in `core/**`, which is held by other executors, so per the standing constraint it is reported rather than attempted. ## Known Stubs None. No placeholder values, no TODOs, no unwired code paths. ## Threat Flags None. No new network endpoint, auth path, file-access pattern or schema change at a trust boundary. The plan installs no packages (T-10-SC: accept); none were added. ## UNVERIFIED — needs hardware Task 3's C-4 checkpoint is now **more** important, not less: with the rootfs stripped and no install-time fallback, the tar listing is the only pre-hardware evidence that the shipped image is identity-free. | Item | Audit ref | What it needs | Command | |---|---|---|---| | Rootfs tar is identity-free after a forced rebuild | **C-4** | ISO build host with podman/docker and disk for a full rootfs rebuild | `UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild`, then the tar listings in `docs/security/KEY-02-ROOTFS-EVIDENCE.md` steps 2/4/5/5b/6 | | The build-time generator assertion actually fires | **C-4** | same build host | `grep 'first-boot secret generators present' ` — evidence doc step 5b | | The self-heal timer ships and is enabled on the target | — | same build host, then a node | evidence doc step 5 (timer present on installer media); `systemctl status archipelago-first-boot-secrets.timer` on a node | | Two nodes flashed from one ISO get different keys | **C-3** | two physical machines | audit §779; both SSH and TLS fingerprints are now equally sharp signals — see the C-3 section of the evidence doc | | The console leg of the failure shout reaches a real screen | — | a real node, or a VM console | force a first-boot failure and observe `/dev/console` | The harness proves the *script* half of self-heal (a failed run followed by a successful run recovers the node and restarts the failed units). It does not and cannot prove systemd's scheduling — that the timer is enabled and actually fires at 5min/15min. That is hardware verification. ## Self-Check - `image-recipe/_archived/build-auto-installer-iso.sh` — FOUND, `bash -n` clean - `tests/first-boot-secrets/run-tests.sh` — FOUND, mode 755, exits 0 with 8 PASS - `docs/security/KEY-02-ROOTFS-EVIDENCE.md` — FOUND, contains `C-4` - `deferred-items.md` — DELETED; its only entry (D1) is fixed, not filed - Commits `21043096`, `408b328c`, `201ef474`, `2efab5f2`, `d9b3a7d5`, `40b77e39` — all FOUND - Generated `Dockerfile.rootfs` rendered before/after the heredoc change and diffed normalised: 190 lines each, only the six comment restorations differ - Single-producer grep: the only executable key-creating invocations in the builder are `openssl req` and `ssh-keygen -A` inside the generator heredoc; every other match is a comment - `git status --porcelain image-recipe/` — clean; `_archived/` not moved or renamed - No file authored by a concurrent agent (`core/archipelago/src/**`, `neode-ui/**`, `.planning/STATE.md`) was staged in any commit ## Self-Check: PASSED