archy/.planning/phases/10-key-material-hardening/10-03-SUMMARY.md

---
phase: 10-key-material-hardening
plan: 03
subsystem: iso-build
tags: [security, iso, first-boot, systemd, ssh-host-keys, tls, machine-id, bash]

requires: []
provides:
  - "Fail-closed first-boot per-device secret generation: the completion marker is written only when both TLS and SSH generation succeeded"
  - "A single producer per secret — gen_tls()/gen_ssh() are the only code in the ISO build that create the TLS keypair and SSH host keys"
  - "Retry-with-backoff inside a single boot, so a transient first-boot condition recovers without a reboot"
  - "archipelago-first-boot-secrets.timer — unattended self-heal every 15 minutes until generation succeeds"
  - "A build-time assertion that fails the ISO build if openssl or ssh-keygen is missing from the rootfs"
  - "Parse-back validation (openssl pkey / openssl x509) before the staging swap, so no service ever reads a truncated artefact"
  - "A durable failure record at /var/lib/archipelago/first-boot-secrets.failed plus console + journal + stderr on failure"
  - "FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF test seams on the generated first-boot script"
  - "An identity-free rootfs tar: no SSH host keys, no TLS keypair, machine-id truncated"
  - "/opt/archipelago/rootfs-identity-stripped build-time provenance marker"
  - "tests/first-boot-secrets/run-tests.sh — 6-case harness driving the shipped heredoc body against a temp root with stubbed generators"
affects: [image-recipe, first-boot, sshd, nginx-tls]

tech-stack:
  added: []
  patterns:
    - "Seam-for-testability over assertion-in-a-comment: FIRST_BOOT_SECRETS_ROOT prefixes every absolute path so the NEGATIVE property (on failure the marker is NOT created) can be forced and asserted. Same move the entropy fix in 8b51b7e2 made for the RNG."
    - "Test the shipped bytes, not a copy: the harness extracts the first-boot script from the builder heredoc between the SECRETSSCRIPT delimiters, so the test and the artefact cannot drift."
    - "Strip identity material in the last Dockerfile layer so fail-closed is structural (no key exists) rather than procedural (a script promises to replace it)."

key-files:
  created:
    - tests/first-boot-secrets/run-tests.sh
    - docs/security/KEY-02-ROOTFS-EVIDENCE.md
  modified:
    - image-recipe/_archived/build-auto-installer-iso.sh

key-decisions:
  - "UNIFY, DO NOT DELETE. The defect in F-03 was never that a second attempt to create a key existed — it was that failure was silent and the marker lied about it. A second attempt is only dangerous when it is an unaudited second PRODUCER with its own idea of success, its own absent retry policy and its own absent failure record. So both secondary producers were folded out (the Dockerfile bake and the installer fallback) leaving one generator per secret, rather than 'keep the fallback' (leaves a silent second source) or 'delete the fallback and accept a dead node' (a false trade between security and UX)."
  - "Fail-closed governs SERVING; self-heal governs RECOVERING. These are separate properties and both must hold. Nothing serves on a key we did not generate; nothing dead-ends waiting for a human at a console."
  - "Self-heal uses a systemd timer, not a sleep loop in the script. A loop would hold a Type=oneshot open for hours and hide the failure from systemctl; the timer plus the service's existing ConditionPathExists=! costs a healthy node nothing and needs no teardown."
  - "The timer's enable uses a hand-written symlink fallback. Every other `chroot systemctl enable` here ends in `2>/dev/null || true`, which would silently drop the self-heal path — the one thing whose absence is invisible until a node is already broken."
  - "Consumers in `failed` state are explicitly restarted on success. try-reload-or-restart is a no-op on a failed unit, so without this a self-healed node would have valid keys on disk and nginx still down — recovery that isn't."
  - "Retry semantics: attempt count equals the number of FIRST_BOOT_SECRETS_BACKOFF entries, and the wait after the final attempt is skipped (a failed last attempt is terminal). With the default `2 8 20` that is 3 attempts at t=0s/2s/10s per generator; the trailing 20 is the ceiling if the list is lengthened. Documented in the script rather than left as a puzzle."
  - "`After=systemd-random-seed.service` added as its own unit line rather than appended to the existing After=local-fs.target, both because systemd accumulates After= lines and because the plan's acceptance criterion greps for exactly that string."
  - "/var/lib/dbus/machine-id is removed only when it is a real file, not when it is the symlink to /etc/machine-id that Debian normally ships. Deleting a live symlink risks a boot-time surprise for no gain; a real copy would be genuine shared state."

requirements-completed: []

coverage:
  - id: D1
    description: "A first-boot secret regeneration that fails does NOT set the completion marker, so the oneshot retries on the next boot (D-05)"
    requirement: KEY-02
    verification:
      - kind: unit
        ref: "tests/first-boot-secrets/run-tests.sh#openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS"
        status: pass
      - kind: other
        ref: "Negative control: moving `touch \"$MARKER\"` back outside the success branch makes that case fail with MARKER-SET-ON-FAILURE (transcript below)"
        status: pass
    human_judgment: false
  - id: D2
    description: "Each generator is retried with backoff within a single boot before the boot is declared failed (D-05)"
    requirement: KEY-02
    verification:
      - kind: unit
        ref: "tests/first-boot-secrets/run-tests.sh#ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)"
        status: pass
    human_judgment: false
  - id: D3
    description: "A terminal failure is loud: console + durable on-disk record + journal, not only a log file nobody reads (D-05)"
    requirement: KEY-02
    verification:
      - kind: unit
        ref: "tests/first-boot-secrets/run-tests.sh#tls-fail case asserts first-boot-secrets.failed exists, names TLS, and stderr carries a FAILED line"
        status: pass
      - kind: other
        ref: "logger + tee -a $ROOT/dev/console emitted by shout(); the console leg cannot be exercised in a temp root and is UNVERIFIED on hardware"
        status: partial
    human_judgment: false
  - id: D4
    description: "The shipped rootfs tar contains no SSH host keys, no TLS private key and no populated machine-id"
    requirement: KEY-04
    verification:
      - kind: other
        ref: "docs/security/KEY-02-ROOTFS-EVIDENCE.md — requires an ISO build host; commands recorded, not yet run"
        status: blocked
    human_judgment: true
  - id: D5
    description: "The regeneration script is exercised by an automated test that fails when the marker is set on a failed run"
    requirement: KEY-02
    verification:
      - kind: unit
        ref: "tests/first-boot-secrets/run-tests.sh — 6/6 PASS; negative control reproduced"
        status: pass
    human_judgment: false
  - id: D6
    description: "Exactly one producer per secret: no second code path anywhere in the ISO build can mint a TLS key or SSH host key with its own accounting"
    requirement: KEY-02
    verification:
      - kind: unit
        ref: "tests/first-boot-secrets/run-tests.sh#single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh"
        status: pass
      - kind: other
        ref: "Negative control: reintroducing the installer's chroot openssl req block turns case 6 red naming the line, and nothing else"
        status: pass
    human_judgment: false
  - id: D7
    description: "A failure self-heals unattended — it never dead-ends a node whose only exit is physical access"
    requirement: KEY-02
    verification:
      - kind: unit
        ref: "tests/first-boot-secrets/run-tests.sh#self-heal: failed run then a later successful run -> key present, marker set, failed units restarted"
        status: pass
      - kind: other
        ref: "archipelago-first-boot-secrets.timer installed + enabled (symlink fallback) — UNVERIFIED on hardware; the harness proves the script half, not systemd's scheduling"
        status: partial
    human_judgment: false
  - id: D8
    description: "The deterministic total-failure cause (missing generator binary) fails the BUILD, not the fleet"
    requirement: KEY-04
    verification:
      - kind: other
        ref: "Rootfs RUN assertion on /usr/bin/openssl and /usr/bin/ssh-keygen; fires during the container build. UNVERIFIED until a build host runs it — see KEY-02-ROOTFS-EVIDENCE.md step 5b"
        status: blocked
    human_judgment: true

duration: 1h
completed: 2026-08-02
status: complete
---

# Phase 10 Plan 03: Fail-closed first-boot secrets + identity-free rootfs — Summary

First-boot per-device secret regeneration now retries with backoff and then fails closed, and
the rootfs tar it repairs no longer carries the fleet-shared SSH host keys, TLS keypair or
machine-id it was silently papering over. Closes the build side of audit finding **F-03**.

> **Task 3 (C-4 build-host evidence) is a blocking checkpoint and is NOT done.** It needs an
> ISO build host. `docs/security/KEY-02-ROOTFS-EVIDENCE.md` carries the exact command sequence
> and is marked UNVERIFIED. Nothing in this plan claims the tar listing was observed.

## What was wrong

`first-boot-secrets.sh` (a heredoc inside `image-recipe/_archived/build-auto-installer-iso.sh`,
which is **live** — `image-recipe/build-debian-iso.sh` execs it) had two fail-open branches that
logged `WARNING: ... keeping baked key` and continued, and `touch "$MARKER"` ran unconditionally
**outside both `if` blocks**. Combined with the unit's
`ConditionPathExists=!/var/lib/archipelago/.secrets-regenerated` and the script's own
`[ -f "$MARKER" ] && exit 0`, one transient failure at first boot left that node on the
image-wide shared SSH host key and TLS private key **permanently and silently** — and the ISO is
a published artefact, so every downloader holds those keys.

## Commits

| Commit | Task | What |
|---|---|---|
| `21043096` | 1 | Fail-closed, retried regeneration + `tests/first-boot-secrets/run-tests.sh` |
| `408b328c` | 2 | Rootfs identity-strip layer + two comment corrections that follow from it |
| `201ef474` | 3 (prep) | `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, marked UNVERIFIED |
| `2efab5f2` | follow-up | Single-producer unification, build-time generator assertion, self-heal timer, 3 new test cases |
| `d9b3a7d5` | follow-up | Quote the `Dockerfile.rootfs` heredoc so comments cannot execute (closes deferred D1) |
| `40b77e39` | follow-up | Refuse to bless a cert minted under an untrustworthy clock; backdate `notBefore` |

Nothing was pushed, tagged, built or deployed, per the execution brief.

## Task 1 — fail-closed regeneration

- `ROOT="${FIRST_BOOT_SECRETS_ROOT:-}"` prefixes every absolute path. Unset in production the
  expansion is empty and behaviour is byte-identical; set, it is what makes the negative
  property assertable at all.
- `retry()` runs each generator up to N times with waits from `FIRST_BOOT_SECRETS_BACKOFF`
  (default `2 8 20`). Staging-then-swap is preserved for both generators, with `.new` files
  removed on failure so no half-keypair is left behind.
- `touch "$MARKER"` now lives inside a `TLS_OK == 1 && SSH_OK == 1` branch. Any other outcome
  writes `/var/lib/archipelago/first-boot-secrets.failed` (timestamp, which generator failed,
  both flags), shouts to console + `logger` + stderr, and `exit 1` so the unit lands in `failed`
  rather than `active`. A later successful boot deletes the record so a recovered node does not
  carry a stale alarm.
- `After=systemd-random-seed.service` added to the unit. A no-op today (no seed file is baked,
  which the audit verified) and correct if one is ever introduced.
- The script header states the operational trade in plain words, including that recovery from a
  terminal failure needs the physical console.

### Harness results (final, all six cases)

```
extracted 236 lines from the builder; bash -n clean
PASS: both generators succeed -> exit 0, marker set, keys swapped in
PASS: openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS
PASS: ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)
PASS: TLS fails every attempt on a stripped root -> NO key, NO marker, non-zero exit, record names TLS
PASS: self-heal: failed run then a later successful run -> key present, marker set, failed units restarted
PASS: single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh

──────── first-boot-secrets summary ────────
passed: 6   failed: 0
```

### Negative control (required by the plan's acceptance criteria)

`touch "$MARKER"` moved back outside the success branch, harness re-run, then reverted:

```
SCRATCH APPLIED: marker touch moved back outside the success branch
--- harness against the fail-open variant ---
extracted 175 lines from the builder; bash -n clean
PASS: both generators succeed -> exit 0, marker set, keys swapped in
FAIL: openssl fails every attempt -> MARKER-SET-ON-FAILURE
    exit=1 root=/tmp/tmp.Ta1YFhHWdi/root-tls-fail
    stderr: ARCHIPELAGO FIRST BOOT FAILED: could not generate this device's TLS key material. ...
PASS: ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)

──────── first-boot-secrets summary ────────
passed: 2   failed: 1
EXIT=1
```

The test fails on exactly the regression it exists to pin, and only that case.

## Task 2 — identity-free rootfs

Final `RUN` layer added to `Dockerfile.rootfs`, after every package install and after the
`openssl req` layer, so nothing regenerates the material afterwards:

- `rm -f /etc/ssh/ssh_host_*` (private keys and `.pub` alike)
- `rm -f` the archipelago TLS key and crt, keeping the `/etc/archipelago/ssl` directory
- `: > /etc/machine-id` (systemd's documented regenerate-on-next-boot state)
- `/var/lib/dbus/machine-id` removed only if it is a real file, not the usual symlink
- writes `/opt/archipelago/rootfs-identity-stripped` listing what it removed, with **no**
  timestamp so RECIPE_HASH reproducibility is unaffected

The `openssl req` layer is deliberately unmodified.

**RECIPE_HASH changed.** The strip layer is inside the hashed region
(`sed -n '/^# STEP 1.../,/^# STEP 2.../p' | grep -c rootfs-identity-stripped` → 1), so the next
build is forced to rebuild the rootfs tar. Task 3's evidence would be meaningless against a
cached tar, and `--rebuild` is specified as well.

`grep -c 'ssh_host'` on the builder went **3 → 6**.

## Deviations from Plan

### 1. [Rule 1 — Bug] Backticks in my own strip-layer comment would have hung every ISO build

- **Found during:** Task 2, self-check of the added block.
- **Issue:** `Dockerfile.rootfs` is written with an **unquoted** heredoc (`<<DOCKERFILE`), so
  backticks in its body are command substitution evaluated by the build shell. Two comment
  lines I wrote contained `` `openssl req` ``. Reproduced in isolation: the heredoc hung for the
  full 2-minute timeout as `openssl req` waited on stdin. `bash -n` is clean on this — syntax
  checking cannot catch it.
- **Fix:** replaced with double quotes, and added an explicit `NOTE:` in the block warning that
  the heredoc is unquoted and backticks must never appear there.
- **Commit:** `408b328c`

### 2. [Rule 1 — Correctness] Script header claim about TLS, twice corrected

- **Found during:** Task 2, after discovering the installer's TLS fallback.
- **Issue:** the Task 1 header claimed "the nginx TLS listener will not start". With the
  installer fallback in place that was false — the web UI would still come up. Shipping a
  confident false statement in a security-critical script is worse than shipping none.
- **First fix (`408b328c`):** narrowed the claim to SSH only, and described the TLS fallback
  honestly as per-install, never image-wide.
- **Second fix (`2efab5f2`):** the fallback is gone, so the original claim is true again for
  both. Restored, with the reasoning attached rather than left implicit. No comment anywhere in
  the builder now implies a TLS fallback exists.

### 3. [Rule 2 — Threat coverage] `/var/lib/dbus/machine-id`

- **Issue:** T-10-26 is machine-id correlation across nodes. The plan named `/etc/machine-id`
  only. If dbus ships a real copy rather than the usual symlink, truncating `/etc/machine-id`
  alone leaves correlated state.
- **Fix:** guarded removal — symlinks are left alone, real files are removed.
- **Commit:** `408b328c`

## Follow-up: unify to a single producer (`2efab5f2`)

The installer's TLS fallback prompted a decision cycle worth recording, because the reasoning
matters more than the outcome.

**The false trade.** The question was framed as "keep the fallback (a second source of keys) or
delete it (a first-boot failure costs the user the web UI, recoverable only at the console)".
Both options were wrong, and the framing was wrong. **The defect in F-03 was never that a second
attempt to create a key existed. It was that failure was silent and the completion marker lied
about it.** A second attempt is only dangerous when it is an unaudited second *producer* —
carrying its own idea of success, its own absent retry policy, its own absent failure record.
So the fix is to unify, not to delete and accept a dead node.

**What shipped:**

1. **One producer per secret.** `gen_tls()` and `gen_ssh()` are the only code in the ISO build
   that create the TLS keypair and the SSH host keys. Two secondary producers were folded out:
   the Dockerfile's `openssl req` layer (which baked a keypair the strip layer deleted moments
   later in the same build) and the installer's "ensure SSL cert exists" block. The invariant is
   checked mechanically, not asserted in prose — case 6 of the harness fails if any executable
   `openssl req` / `ssh-keygen -A` invocation appears outside the generator heredoc.

2. **The deterministic failure is caught at build time.** The one realistic way generation fails
   on every retry forever is a missing generator binary, and that is deterministic — no retry or
   reboot fixes it. A rootfs `RUN` layer now fails the build if `/usr/bin/openssl` or
   `/usr/bin/ssh-keygen` is missing or non-executable. **The build already guaranteed these**
   (`openssl` and `openssh-server` are both in the package list, and `openssh-server`
   hard-depends `openssh-client`, which ships `ssh-keygen`), so this is cheap insurance rather
   than a fix. It earns its place the first time someone edits that package list.

3. **Failure self-heals; it never dead-ends.** `archipelago-first-boot-secrets.timer`
   (`OnBootSec=5min`, `OnUnitActiveSec=15min`) re-runs the service until it succeeds. The
   service's existing `ConditionPathExists=!` makes every trigger a no-op once the marker
   exists, so a healthy node pays nothing and no teardown is needed. Two details that would have
   made this theatre if missed:
   - `chroot systemctl enable` can fail silently, and every other enable in this file ends in
     `|| true`. The timer's enable has a hand-written symlink fallback, because the absence of
     self-heal is invisible until a node is already broken.
   - `try-reload-or-restart` is a **no-op on a failed unit**. Without special handling, a
     self-healed node would have valid keys on disk and nginx still down. Consumers found in
     `failed` are now explicitly restarted (`--no-block`, to avoid a boot-transaction deadlock
     at first boot, where we are ordered `Before=` them).

4. **Never serve a bogus key.** `gen_tls` now parses both halves back (`openssl pkey`,
   `openssl x509`) before the staging swap, so a truncated or half-written artefact is never
   what nginx reads. Fail-closed governs *serving*; retry-and-self-heal governs *recovering*.
   They are different properties and both hold.

### Negative controls for the three new cases

Each defect was reintroduced, the suite run, and the defect reverted. Each lights up **exactly
one** case — a test that goes red for several reasons at once is not pinning any of them.

**Control A — reintroduce a fallback-style key creation on the failure path** (the deleted
installer block's behaviour, moved into the script):

```
FAIL: TLS fails every attempt on a stripped root -> TLS-KEY-EXISTS-AFTER-FAILURE TLS-CRT-EXISTS-AFTER-FAILURE
passed: 5   failed: 1
```

*(First run of this control also reddened case 5, because case 5's run-1 block redundantly
re-asserted case 4's property. That assertion was removed — case 5 now tests recovery only —
and the control re-run to confirm it is isolated. The transcript above is the re-run.)*

**Control B — dead-end a node that has already failed once** (`exit 0` early if the failure
record exists, a plausible "don't retry a known-bad node" optimisation):

```
FAIL: self-heal -> run2-marker-missing run2-key-missing run2-crt-missing run2-stale-failure-record run2-did-not-restart-failed-nginx
passed: 5   failed: 1
```

**Control C — reintroduce the installer's `chroot ... openssl req` block verbatim:**

```
FAIL: single-producer invariant -> SECOND-PRODUCER-at-line-3586
    generator heredoc spans lines 1713-1950 of image-recipe/_archived/build-auto-installer-iso.sh
passed: 5   failed: 1
```

All three reverted; suite back to 6/6.

## Residual operational risk — stated plainly

**A machine on which secret generation can never succeed ends up with no SSH host key and no
TLS key. sshd will not start, nginx will not serve the web UI, and that node needs physical
console access.** That is the honest worst case and it is not softened anywhere in the code
comments either.

What shrinks it to genuinely-broken-hardware:

- **The deterministic cause is gone before shipping.** A missing `openssl` or `ssh-keygen` fails
  the ISO build, so it cannot reach a node.
- **Transient causes are absorbed.** Three attempts with backoff inside the boot (proven by
  harness case 3, which shows a generator failing twice and succeeding on the third), then every
  15 minutes on the timer, then again on every boot — indefinitely, because the marker is never
  written on failure.
- **Recovery completes itself.** On a later success the script restarts the units that refused
  to start, so the node comes back without a reboot and without a human (harness case 5).

What is left is a machine where `openssl` or `ssh-keygen` is present but cannot ever produce a
key — a disk that is permanently full, or failing hardware. On that machine the node refuses to
serve rather than serving on a key nobody generated, which is the trade this phase exists to
make. It says so on the console, in the journal, and in
`/var/lib/archipelago/first-boot-secrets.failed`.

## Follow-up: quote the Dockerfile heredoc (`d9b3a7d5`) — closes deferred D1

`cat > "$WORK_DIR/Dockerfile.rootfs" <<DOCKERFILE` was **unquoted**, so the build shell
performed command substitution on the Dockerfile body: a backtick inside a Dockerfile *comment*
was executed on the build host and its output spliced into the generated file. Six comments did
this, and one of them ran `systemctl start archipelago-fips.service` against the build machine
on every ISO build.

**Boundary checked before editing.** Only lines inside the heredoc body are at risk. The other
backticked comments in this file (`:264`, `:809`, `:1188`, `:1289`, `:1506`, `:1605`, `:3597`,
`:3651`) are ordinary shell comments outside any unquoted heredoc, plus one inside the *quoted*
`SECRETSSCRIPT` heredoc — none were ever evaluated, and none were touched.

**Fixed the class, not the instances.** The body needs exactly four build-time values, all
package names (`LINUX_IMAGE_PKG`, `GRUB_EFI_PKG`, `GRUB_EFI_SIGNED_PKG`, `GRUB_PC_PKG`), on four
consecutive lines — so quoting was entirely practical. The heredoc is split into
`DOCKERFILE_HEAD` and `DOCKERFILE_TAIL`, both quoted, with one explicit `printf` interpolating
those four names between them. Escapes that existed *only* because the heredoc was unquoted were
undone in the same pass: six trailing `\\` → `\` (Docker line continuations) and four `\$` → `$`
(RUN arguments reach the shell verbatim; Docker does not substitute variables in RUN).

**Substance verified by rendering, not by inspection.** The generated Dockerfile was rendered
before and after with identical inputs and diffed *normalised* (continuations joined, whitespace
collapsed). Both are 190 normalised lines and the only differences are the six comments regaining
their text — every instruction byte-identical:

```
< # the archipelago backend calls
> # the archipelago backend calls `systemctl start archipelago-fips.service`
< # fips-gateway is gated behind the Cargo feature (depends on
> # fips-gateway is gated behind the `gateway` Cargo feature (depends on
```

**Case 7** asserts every heredoc writing `Dockerfile.rootfs` has a quoted delimiter, and when one
does not, reports which body lines would execute. The assertion is on the **delimiter, not on
backticks** — with quoting a backticked comment is legal, and six of them are back in the body on
purpose; flagging backticks would flag a non-bug and fail on the very comments this restored.

Controls:

```
Control D — unquote the delimiter (the real regression):
FAIL: Dockerfile heredoc quoting -> UNQUOTED-DELIMITER-at-line-287 would-execute-at-lines:317,318
passed: 6   failed: 1

Control E — add a backticked comment, delimiter still quoted:
PASS: Dockerfile heredoc delimiters are quoted — a backticked comment cannot execute
passed: 7   failed: 0
  and it renders intact:
  175:# Control E: a backticked `systemctl start archipelago-fips.service` comment
```

Control E is the more informative of the two: the backtick that used to be a build-host RCE is
now inert and renders as written. That is what "fixed the class" means, and it is why a bare
backtick reintroduction correctly reddens nothing.

`deferred-items.md` held D1 as its only entry and has been **deleted** — nothing was left that is
genuinely out of scope.

## Follow-up: untrustworthy clock at cert-minting time (`40b77e39`)

The failure fail-closed cannot catch, **because generation succeeds**. This unit runs before time
has synced; `openssl req -x509` stamps `notBefore` from whatever the clock says. Dead RTC or flat
CMOS battery → clock ahead gives "not yet valid" (harder to diagnose than a self-signed warning),
clock behind gives an already-expired cert once time syncs. The marker was then set and never
revisited: a node permanently serving a cert nothing accepts.

**Finding, established rather than assumed:** this image does **not** use `systemd-timesyncd`. It
installs and enables **chrony** (`:388`, `:575`), and `chrony-wait.service` — the unit that is
`Before=time-sync.target` — is **not** enabled. So `time-sync.target` is inert here and ordering
`After=` it would buy nothing. Enabling `chrony-wait` to make it meaningful would stall boot
behind NTP on a node with no network, and these nodes are routinely offline at first boot.

**Decision: no ordering change.** Not deadlocking boot outranks cert-date elegance (constraint 3).
Fixed locally instead:

1. **Backdate `notBefore` by 24h** so ordinary node/client skew cannot invalidate a fresh cert.
   This does not weaken a self-signed cert — `notBefore` is not a security control here.
   `-not_before`/`-not_after` arrived in OpenSSL 3.5 and the rootfs is `debian:trixie` which
   ships it, but the capability is **probed, not assumed**: guessing wrong would fail every
   attempt and brick the node, the exact outcome all of this exists to prevent. Without the flags
   we do not backdate, and rule 2 still covers the dangerous case.
2. **Refuse to bless a cert dated outside a plausible window** (2026-01-01 … 2056-01-01). The
   material stays installed — the node is usable, sshd comes up — but the dates are recorded as
   `failed=cert-dates` and the cert is regenerated automatically once time syncs.

Generation is now driven by **need** rather than "is the marker absent", and
`ConditionPathExists=!` was removed from the unit so a node that already completed can still be
re-examined. Skipping the unit is precisely how such a node would stay broken forever. On a
healthy node the script exits in milliseconds.

**Anti-spin is one condition:** a date-driven regeneration happens *only* when the clock is
currently plausible. A node whose clock is still wrong re-checks and mints nothing.

### Regression I introduced and caught

Driving generation purely by content made `needs_ssh()` false whenever *any* host key existed —
which would have left an image-baked, fleet-shared host key in place forever. **That is F-03
reopened.** The marker check is back in both `needs_` functions. Case 1 — which prestages a baked
key and asserts it was replaced — is what caught it.

### Controls

```
Control F — bless the cert regardless of clock (the pre-fix behaviour):
FAIL: wrong clock -> run1-BAD-DATES-NOT-RECORDED
passed: 7   failed: 1

Control G — remove the anti-spin guard:
FAIL: wrong clock -> SPINNING-reminted-while-clock-still-wrong(1->2)
passed: 7   failed: 1
```

**Control G first passed against a deliberately broken guard**, which was a flaw in my test, not
in the fix: the assertion compared certificate dates, and a re-mint under a frozen fake clock
produces a byte-identical `notBefore`. Dates cannot distinguish "left alone" from "regenerated
again". The assertion now counts `openssl req` invocations, which can — and only then did the
control redden. Worth recording as the second time in this plan that a first-draft assertion
looked green for the wrong reason.

### Not covered here

Nodes already deployed from earlier ISOs **never receive this script** — it is installed by the
installer, not shipped by OTA. Fleet remediation for those nodes is 10-04/OTA work in `core/**`,
which is held by other executors, so per the standing constraint it is reported rather than
attempted.

## Known Stubs

None. No placeholder values, no TODOs, no unwired code paths.

## Threat Flags

None. No new network endpoint, auth path, file-access pattern or schema change at a trust
boundary. The plan installs no packages (T-10-SC: accept); none were added.

## UNVERIFIED — needs hardware

Task 3's C-4 checkpoint is now **more** important, not less: with the rootfs stripped and no
install-time fallback, the tar listing is the only pre-hardware evidence that the shipped image
is identity-free.

| Item | Audit ref | What it needs | Command |
|---|---|---|---|
| Rootfs tar is identity-free after a forced rebuild | **C-4** | ISO build host with podman/docker and disk for a full rootfs rebuild | `UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild`, then the tar listings in `docs/security/KEY-02-ROOTFS-EVIDENCE.md` steps 2/4/5/5b/6 |
| The build-time generator assertion actually fires | **C-4** | same build host | `grep 'first-boot secret generators present' <build log>` — evidence doc step 5b |
| The self-heal timer ships and is enabled on the target | — | same build host, then a node | evidence doc step 5 (timer present on installer media); `systemctl status archipelago-first-boot-secrets.timer` on a node |
| Two nodes flashed from one ISO get different keys | **C-3** | two physical machines | audit §779; both SSH and TLS fingerprints are now equally sharp signals — see the C-3 section of the evidence doc |
| The console leg of the failure shout reaches a real screen | — | a real node, or a VM console | force a first-boot failure and observe `/dev/console` |

The harness proves the *script* half of self-heal (a failed run followed by a successful run
recovers the node and restarts the failed units). It does not and cannot prove systemd's
scheduling — that the timer is enabled and actually fires at 5min/15min. That is hardware
verification.

## Self-Check

- `image-recipe/_archived/build-auto-installer-iso.sh` — FOUND, `bash -n` clean
- `tests/first-boot-secrets/run-tests.sh` — FOUND, mode 755, exits 0 with 8 PASS
- `docs/security/KEY-02-ROOTFS-EVIDENCE.md` — FOUND, contains `C-4`
- `deferred-items.md` — DELETED; its only entry (D1) is fixed, not filed
- Commits `21043096`, `408b328c`, `201ef474`, `2efab5f2`, `d9b3a7d5`, `40b77e39` — all FOUND
- Generated `Dockerfile.rootfs` rendered before/after the heredoc change and diffed normalised:
  190 lines each, only the six comment restorations differ
- Single-producer grep: the only executable key-creating invocations in the builder are
  `openssl req` and `ssh-keygen -A` inside the generator heredoc; every other match is a comment
- `git status --porcelain image-recipe/` — clean; `_archived/` not moved or renamed
- No file authored by a concurrent agent (`core/archipelago/src/**`, `neode-ui/**`,
  `.planning/STATE.md`) was staged in any commit

## Self-Check: PASSED