---
phase: 10-key-material-hardening
plan: 03
type: execute
wave: 1
depends_on: []
files_modified:
- image-recipe/_archived/build-auto-installer-iso.sh
- tests/first-boot-secrets/run-tests.sh
- docs/security/KEY-02-ROOTFS-EVIDENCE.md
autonomous: false
requirements: [KEY-02, KEY-04]
must_haves:
truths:
- "A first-boot secret regeneration that fails does NOT set the completion marker, so the oneshot retries on the next boot (D-05)"
- "Each generator is retried with backoff within a single boot before the boot is declared failed (D-05)"
- "A terminal failure is loud: it reaches the console and a durable on-disk failure record, not only a log file nobody reads (D-05)"
- "The shipped rootfs tar contains no SSH host keys, no TLS private key and no populated machine-id, so a regeneration failure degrades to 'no key, service refuses to start' rather than 'fleet-shared key, silently'"
- "The regeneration script is exercised by an automated test that fails when the marker is set on a failed run"
artifacts:
- path: "image-recipe/_archived/build-auto-installer-iso.sh"
provides: "Fail-closed, retried first-boot secret regeneration and an identity-free rootfs tar"
contains: "FIRST_BOOT_SECRETS_ROOT"
- path: "tests/first-boot-secrets/run-tests.sh"
provides: "Automated harness that extracts the generated script and drives it with stubbed generators"
min_lines: 60
- path: "docs/security/KEY-02-ROOTFS-EVIDENCE.md"
provides: "Recorded build-host evidence for audit checklist item C-4"
contains: "C-4"
key_links:
- from: "tests/first-boot-secrets/run-tests.sh"
to: "image-recipe/_archived/build-auto-installer-iso.sh"
via: "extracts the first-boot-secrets.sh heredoc body from the builder and executes it against a temp root"
pattern: "SECRETSSCRIPT"
- from: "image-recipe/_archived/build-auto-installer-iso.sh"
to: "docs/security/KEY-02-ROOTFS-EVIDENCE.md"
via: "the Dockerfile strip step is what the C-4 tar listing proves"
pattern: "ssh_host"
---
Close F-03 (High) on the build side: make first-boot per-device secret regeneration **retry with
backoff and then fail closed** (D-05), and remove the fleet-shared identity material from the
rootfs tar so a failure degrades to "no key" instead of "everyone's key".
Purpose: today both regeneration branches log a warning and continue, and
`touch "$MARKER"` runs unconditionally outside both `if` blocks
(`image-recipe/_archived/build-auto-installer-iso.sh:1647`, `:1659`, `:1663`). Combined with
`ConditionPathExists=!/var/lib/archipelago/.secrets-regenerated` (`:1605`) and the script's own
`[ -f "$MARKER" ] && exit 0` (`:1625`), one transient failure leaves that node on the
**image-wide shared** SSH host key and TLS private key permanently and silently — and the ISO is
a published artefact, so anyone who downloads it holds those keys.
Output: a fail-closed regeneration script with a real automated test, an identity-free rootfs,
and recorded C-4 build-host evidence.
**`image-recipe/_archived/` is LIVE.** `image-recipe/build-debian-iso.sh:19-40` copies it to a
temp path, rewrites its relative paths and `exec`s it. Do not relocate, rename or tidy it — the
audit records that treating it as dead would have hidden F-03 entirely.
@$HOME/.claude/gsd-core/workflows/execute-plan.md
@$HOME/.claude/gsd-core/templates/summary.md
@.planning/PROJECT.md
@.planning/STATE.md
@.planning/phases/10-key-material-hardening/10-CONTEXT.md
@docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md
@CLAUDE.md
`RECIPE_HASH` (`build-auto-installer-iso.sh:265`) hashes only the region between
`# STEP 1: Build complete root filesystem` (line 252) and `# STEP 2: Build minimal installer`
(line 732), and the rootfs tar is rebuilt only when that hash changes. Consequences the executor
must plan around:
- Task 1 edits the first-boot script heredoc at ~1590-1670, which is in STEP 3 — **outside** the
hashed region. It does not and should not force a rootfs rebuild; it is installer-side content.
- Task 2 edits the Dockerfile inside STEP 1, so the hash changes and the next build rebuilds the
rootfs automatically. That is required for Task 3's C-4 evidence to mean anything: a cached
tar would still contain the baked keys and the check would fail for the wrong reason.
Task 1: Fail-closed, retried first-boot regeneration — proven end to end by a real test
image-recipe/_archived/build-auto-installer-iso.sh, tests/first-boot-secrets/run-tests.sh
- image-recipe/_archived/build-auto-installer-iso.sh (lines 1590-1675 — the unit definition and the whole first-boot-secrets.sh heredoc being rewritten)
- image-recipe/_archived/build-auto-installer-iso.sh (lines 2580-2600 and 3330-3345 — where the script and unit are installed and enabled, so the executor can confirm nothing else needs changing)
- docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (finding F-03 and remediation R-02)
- tests/lifecycle/TESTING.md (house conventions for a repo test harness — exit codes, output shape)
- scripts/first-boot-containers.sh (house style for a first-boot script on this project)
Rewrite the `first-boot-secrets.sh` heredoc body inside
`image-recipe/_archived/build-auto-installer-iso.sh` (currently lines ~1616-1665) so it is
retried-then-fail-closed, and add one testability seam.
**Testability seam (required, and the reason the rest of this task is verifiable at all):**
introduce `ROOT="${FIRST_BOOT_SECRETS_ROOT:-}"` at the top and prefix every absolute path with
`$ROOT` — `$ROOT/etc/archipelago/ssl`, `$ROOT/etc/ssh`, `$ROOT/var/lib/archipelago`,
`$ROOT/var/log`. With the variable unset the expansion is empty and production behaviour is
byte-identical to today. This is the same move the audit made for the RNG: create a seam so the
property can be tested, rather than asserting it in a comment.
**Retry with backoff (D-05):** wrap each generator in a loop of 3 attempts with sleeps of 2, 8
and 20 seconds between them. Track `TLS_OK` and `SSH_OK` as `0`/`1`. Keep the existing
staging-then-swap structure for both — generate to `.new` / a `mktemp -d` staging tree and only
swap on success — because that is what guarantees the node is never left mid-swap.
**Fail closed (D-05):** move `touch "$MARKER"` inside a branch that requires
`TLS_OK = 1 && SSH_OK = 1`. On any other outcome: do not create the marker (so
`ConditionPathExists=!` lets the oneshot run again on the next boot), write a durable failure
record to `$ROOT/var/lib/archipelago/first-boot-secrets.failed` containing the timestamp and
which generator failed, emit the failure to the console with `tee -a /dev/console` (guarded so a
missing `/dev/console` in a test root cannot itself fail the script) and to the journal via
`logger -t archipelago-first-boot-secrets`, and `exit 1` so the unit lands in `failed` rather
than `active`. Delete the `first-boot-secrets.failed` record on a successful run so a node that
recovers on its second boot does not carry a stale alarm.
**Unit ordering:** add `After=systemd-random-seed.service` to
`archipelago-first-boot-secrets.service` (line ~1603) alongside the existing
`After=local-fs.target`. It is a no-op today — no seed file is baked, which the audit verified —
and correct if one is ever introduced. Leave `DefaultDependencies=no`,
`Before=ssh.service nginx.service archipelago.service` and the `ConditionPathExists` line as they
are.
State the operational consequence in a comment at the top of the script, in plain words: after
Task 2 strips the baked material, a terminal failure means the node has no SSH host key and no
TLS key, so `sshd` and the nginx TLS listener will not start and recovery requires the physical
console. That is the deliberate trade D-05 chose over running on fleet-shared keys, and the next
person to read this script deserves to see it stated rather than discover it.
Then create `tests/first-boot-secrets/run-tests.sh` (executable, `set -euo pipefail`). It
extracts the heredoc body from the builder with `awk` between the `SECRETSSCRIPT` delimiters,
writes it to a temp file, and runs it three times against a fresh temp root with a stub `PATH`
that shadows `openssl`, `ssh-keygen`, `systemctl` and `logger`:
- **both succeed** — assert exit 0, marker file present, no `first-boot-secrets.failed`, and the
swapped TLS key and host keys present at their final paths.
- **openssl fails every attempt** — assert exit non-zero, marker file ABSENT,
`first-boot-secrets.failed` present and naming TLS, and no `.new` leftovers.
- **ssh-keygen fails twice then succeeds** — assert exit 0 and marker present, proving the
backoff retry actually recovers rather than just delaying a failure. Have the stub use a
counter file so the third invocation succeeds, and shorten the waits for the test by driving
the sleeps through a `FIRST_BOOT_SECRETS_BACKOFF` variable defaulting to `2 8 20`.
Print a `PASS`/`FAIL` line per case and exit non-zero if any case fails.
bash tests/first-boot-secrets/run-tests.sh
- `bash tests/first-boot-secrets/run-tests.sh` exits 0 and prints three `PASS` lines.
- The failure case asserts the marker is absent; the executor records a scratch run with `touch "$MARKER"` moved back outside the success branch, which MUST make that case fail — pasted into the SUMMARY, then reverted.
- `bash -n` is clean on the builder: `bash -n image-recipe/_archived/build-auto-installer-iso.sh`.
- Extracting the heredoc and running `bash -n` on the extracted body is clean (the harness does this as its first step).
- `grep -c 'FIRST_BOOT_SECRETS_ROOT' image-recipe/_archived/build-auto-installer-iso.sh` is at least 1, and `grep -c 'After=systemd-random-seed.service' image-recipe/_archived/build-auto-installer-iso.sh` is exactly 1.
- `image-recipe/_archived/` is not moved, renamed, or referenced from a new location: `git status --porcelain image-recipe/` shows only a modification to `build-auto-installer-iso.sh`.
A failed regeneration leaves no marker, writes a durable failure record, reaches the console, and exits non-zero — and a transient failure recovers via backoff within the same boot, all proven by an automated harness.
Task 2: Strip fleet-shared identity material from the rootfs tar at build time
image-recipe/_archived/build-auto-installer-iso.sh
- image-recipe/_archived/build-auto-installer-iso.sh (lines 252-272 for the RECIPE_HASH cache condition; 330-355 for the package list that installs openssh-server; 455-470 for the baked TLS keypair; 710-726 for the container export that becomes the shipped tar)
- docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (finding F-03 and remediation R-03; and the ARCHY-3 table row on machine-id, which is the remaining UNVERIFIED item this task also closes on the build side)
- image-recipe/configs/nginx-archipelago.conf (confirms nginx's TLS server block depends on /etc/archipelago/ssl/archipelago.key, i.e. what "fail closed" actually costs)
Add a final `RUN` layer to the rootfs `Dockerfile.rootfs` heredoc inside STEP 1 (after the TLS
generation at ~line 463-469 and after every package install, so nothing regenerates them
afterwards) that removes the identity material Debian's `openssh-server` postinst and the
`openssl req` step bake into the shared image:
- delete every `/etc/ssh/ssh_host_*` file (private keys and `.pub` alike),
- delete `/etc/archipelago/ssl/archipelago.key` and `/etc/archipelago/ssl/archipelago.crt`,
keeping the `/etc/archipelago/ssl` directory itself so the first-boot script's `mkdir -p` and
the later swap have somewhere to land,
- truncate `/etc/machine-id` to zero length (`: > /etc/machine-id`), which is systemd's
documented "generate on next boot" state and is what makes two nodes flashed from one ISO have
different machine-ids.
Keep the `openssl req` step where it is rather than deleting it — leaving it means the build
still proves `openssl` is present and the SAN template still lives next to the code that uses it;
the strip layer is what makes the output non-shared. Add a comment on the strip layer naming
F-03 and stating that its purpose is to make a first-boot regeneration failure degrade to
"no key, service refuses to start" instead of "fleet-shared key, silently".
Also write a build-time provenance line: have the strip layer create
`/opt/archipelago/rootfs-identity-stripped` containing the strings it removed, so a node can
answer after the fact whether its rootfs came from a stripped build. Do not put a build
timestamp in it — that would defeat the reproducibility the RECIPE_HASH cache depends on.
Note in the SUMMARY that this edit is inside the hashed region and therefore forces the next
build to rebuild the rootfs tar, which Task 3 depends on.
bash -n image-recipe/_archived/build-auto-installer-iso.sh && sed -n '/^# STEP 1: Build complete root filesystem/,/^# STEP 2: Build minimal installer/p' image-recipe/_archived/build-auto-installer-iso.sh | grep -c 'rootfs-identity-stripped'
- `bash -n image-recipe/_archived/build-auto-installer-iso.sh` exits 0.
- The strip layer is inside the hashed region: the `sed` range extraction above finds `rootfs-identity-stripped` at least once, so the next build invalidates the cached tar.
- The strip layer removes all four artefact classes; `grep -c 'ssh_host' image-recipe/_archived/build-auto-installer-iso.sh` increases by at least 1 relative to the pre-plan count, which the executor records in the SUMMARY.
- `bash tests/first-boot-secrets/run-tests.sh` still exits 0 (Task 1's harness must not regress).
- The `openssl req` block at ~line 463 is still present and unmodified.
The rootfs tar the installer extracts onto every disk carries no SSH host keys, no TLS private key and no populated machine-id, and the next build is forced to rebuild it.
Task 3: C-4 — prove the shipped rootfs tar is identity-free on the build host
An ISO build host with the image-recipe prerequisites (podman or docker, and enough disk for a full rootfs rebuild) is available; the repo checkout on it contains Task 1 and Task 2's commits.
docs/security/KEY-02-ROOTFS-EVIDENCE.md
- docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (section 6, checklist item C-4 — the exact tar listing and its expected result, which THIS plan deliberately inverts)
- image-recipe/_archived/build-auto-installer-iso.sh (Task 2 output — the strip layer whose effect is being measured)
- image-recipe/build-debian-iso.sh (lines 15-40 — the wrapper that execs the archived builder, and the `UNBUNDLED=1` convention from CLAUDE.md/project memory)
Claude prepares the exact command sequence and, after the operator responds, records the raw tar
listing and verdict into `docs/security/KEY-02-ROOTFS-EVIDENCE.md` under a `## C-4 — rootfs tar
contents` heading, together with the build-host label, the builder commit sha and the RECIPE_HASH
observed.
The expectation is deliberately the INVERSE of the audit's. The audit expected SSH host keys and
the TLS key **present** (they were baked) and recorded that "anything else changes F-03's
severity". After Task 2 they must be **absent** — so the audit's stated expectation is now the
failure condition. Say that explicitly in the evidence document so a future reader comparing the
two does not conclude the check regressed.
A rootfs Dockerfile that strips baked SSH host keys, the TLS keypair and machine-id from the shared image, plus a fail-closed regeneration script that recreates them per node.
1. On the build host, from the repo root, force a full rebuild so the cached tar cannot mask the change:
`UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild`
2. Locate the produced tar (the builder prints its path; it is the `$ROOTFS_TAR` it exported) and list the identity artefacts:
`tar -tvf /archipelago-rootfs.tar | grep -E 'etc/ssh/ssh_host|etc/machine-id|var/lib/systemd/random-seed|archipelago/ssl/archipelago'`
3. Expected after this plan: no `etc/ssh/ssh_host_*` entries at all; no `archipelago/ssl/archipelago.key` or `.crt`; no `var/lib/systemd/random-seed`; `etc/machine-id` present with size 0.
4. Confirm the provenance file rode along: `tar -tvf /archipelago-rootfs.tar | grep rootfs-identity-stripped`
5. Confirm the first-boot pieces are still shipped onto the installer media:
`ls -l /installer-iso/archipelago/scripts/first-boot-secrets.sh /installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service`
6. Paste the full output of steps 2, 3, 4 and 5.
- The step-2 listing shows zero `etc/ssh/ssh_host` entries and zero `archipelago/ssl/archipelago.key` entries.
- `etc/machine-id` appears with size `0`, or is recorded as absent with that stated explicitly — either satisfies "not shared", and the evidence document must say which was observed rather than generalising.
- `var/lib/systemd/random-seed` is absent, re-confirming the audit's negative finding against the rebuilt tar rather than inheriting it.
- Steps 4 and 5 both succeed, proving the strip layer ran and the regeneration script is still installed — a stripped rootfs with no regeneration script would be a brick, and this criterion is what catches that.
- `docs/security/KEY-02-ROOTFS-EVIDENCE.md` marks audit item C-4 as VERIFIED with the date, build-host label and builder commit sha, and states the inverted expectation explicitly.
Paste the tar listings from steps 2-5, then type "approved" — or describe what was still present.
Audit item C-4 is no longer UNVERIFIED, and the recorded evidence shows the shipped rootfs is identity-free while the regeneration path is still installed.
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Published ISO -> any downloader | The ISO is a public artefact. Anything identity-shaped inside it is known to every attacker who fetches it. This is the boundary F-03 crosses. |
| Shared rootfs tar -> every flashed node | `tar -xf "$ROOTFS_TAR" -C /mnt/target` (`:2303`) puts a byte-identical filesystem on every disk. |
| First boot -> network-facing services | `archipelago-first-boot-secrets.service` runs `Before=ssh.service nginx.service archipelago.service`; whatever it leaves behind is what those services present to the network. |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-10-21 | Spoofing | Fleet-shared SSH host key from the published ISO enables undetectable host impersonation | high | mitigate | Task 2 strips the baked host keys; Task 1 makes regeneration fail closed so a failure cannot silently restore the shared state |
| T-10-22 | Information disclosure | Fleet-shared TLS private key from the published ISO enables transparent MITM of the web UI | high | mitigate | Task 2 strips the baked TLS keypair; Task 1's staging-then-swap keeps the swap atomic |
| T-10-23 | Tampering | The completion marker is set on a failed run, so the failure is permanent and unretried (`:1663`) | high | mitigate | Task 1 moves `touch "$MARKER"` inside a both-succeeded branch and writes a durable failure record instead |
| T-10-24 | Denial of service | Fail-closed leaves a node with no SSH and no TLS after a terminal failure, unrecoverable remotely | high | mitigate | Three attempts with 2/8/20s backoff within the boot, then retry on every subsequent boot because the marker is absent; the trade is stated in the script header and is D-05's explicit choice; physical console recovery exists on these nodes |
| T-10-25 | Repudiation | The only record of a failure is a log file that surfaces nowhere | medium | mitigate | Task 1 adds `/var/lib/archipelago/first-boot-secrets.failed`, a console write and a `logger` line; surfacing it in the daemon's status output is 10-04's job |
| T-10-26 | Spoofing | Correlated `machine-id` across nodes flashed from one ISO | medium | mitigate | Task 2 truncates `/etc/machine-id` so systemd regenerates per node; the observed result is recorded in Task 3 rather than assumed |
| T-10-27 | Tampering | A cached rootfs tar masks the strip layer, so C-4 passes against stale output | medium | mitigate | The strip layer is inside the RECIPE_HASH region and Task 3 additionally passes `--rebuild`; the acceptance criterion checks the extraction range, not just the file |
| T-10-28 | Denial of service | A stripped rootfs ships without the regeneration script, bricking every flashed node | high | mitigate | Task 3 step 5 explicitly checks that `first-boot-secrets.sh` and its unit are present on the installer media, and that check is an acceptance criterion |
| T-10-SC | Tampering | npm/pip/cargo installs | low | accept | This plan installs no packages; it edits a shell builder and adds a bash test harness. The Debian package list in the rootfs Dockerfile is not modified. Executor MUST halt and raise a checkpoint if a package addition appears necessary. |
## Artifacts this plan produces
**Modified:** `image-recipe/_archived/build-auto-installer-iso.sh`
| Symbol | Kind | Contract |
|---|---|---|
| `FIRST_BOOT_SECRETS_ROOT` | env var read by the generated `first-boot-secrets.sh` | path prefix for every absolute path; unset in production, set by the test harness |
| `FIRST_BOOT_SECRETS_BACKOFF` | env var read by the generated script | space-separated backoff seconds; default `2 8 20` |
| `/var/lib/archipelago/first-boot-secrets.failed` | new on-disk file | durable failure record: timestamp plus which generator failed; deleted on a later successful run |
| `/opt/archipelago/rootfs-identity-stripped` | new on-disk file | build-time provenance: the artefact classes removed from the rootfs |
| `/var/lib/archipelago/.secrets-regenerated` | existing marker, contract changed | now written ONLY when both TLS and SSH regeneration succeeded |
| `After=systemd-random-seed.service` | unit ordering | added to `archipelago-first-boot-secrets.service` |
**New file:** `tests/first-boot-secrets/run-tests.sh` (mode 755) — three cases: both-succeed,
openssl-always-fails, ssh-keygen-fails-twice-then-succeeds. Exit 0 only if all three pass.
**New file:** `docs/security/KEY-02-ROOTFS-EVIDENCE.md` — heading `## C-4 — rootfs tar contents`.
- `bash tests/first-boot-secrets/run-tests.sh` exits 0 with three `PASS` lines.
- `bash -n image-recipe/_archived/build-auto-installer-iso.sh` is clean.
- Task 3's checkpoint resolved with a pasted tar listing from a `--rebuild` build.
- Commit stages only `image-recipe/_archived/build-auto-installer-iso.sh`,
`tests/first-boot-secrets/run-tests.sh` and `docs/security/KEY-02-ROOTFS-EVIDENCE.md` by
explicit path — never `git add -A`; another agent shares this tree.
- A failed regeneration leaves no completion marker, writes a durable failure record, reaches the
console and exits non-zero — pinned by an automated test that fails if the marker moves back out.
- A transient failure recovers via backoff inside the same boot.
- The shipped rootfs tar contains no SSH host keys, no TLS private key and no populated
machine-id, verified against a forced rebuild on a real build host.
- The regeneration script and its unit are still installed onto the installer media.
- Audit item C-4 is recorded as VERIFIED with the inverted expectation stated explicitly.