archy/docs/security/KEY-02-ROOTFS-EVIDENCE.md

KEY-02 — build-host evidence for the rootfs identity strip

Audit item: C-4 of docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md (§868), which belongs to finding F-03 (fail-open, never-retried first-boot secret regeneration over a fleet-shared rootfs).

Status: ⛔ UNVERIFIED — awaiting a run on a real ISO build host.

The code change is committed and unit-tested; the tar listing that proves its effect on a real build has not been produced yet, because it requires a build host with podman/docker and enough disk for a full rootfs rebuild. Do not read anything below the "Result" heading as a passing check until it is filled in.

Field	Value
Builder commit (Task 1)	21043096 — fail-closed first-boot regeneration
Builder commit (Task 2)	408b328c — rootfs identity strip
Builder commit (follow-up)	single-producer unification, build-time generator assertion, self-heal timer
Builder file	image-recipe/_archived/build-auto-installer-iso.sh (LIVE; image-recipe/build-debian-iso.sh execs it)
Build host	to be recorded
Date run	to be recorded
RECIPE_HASH observed	to be recorded — read it from the stamp file, see the caveat below

---

The expectation is deliberately INVERTED relative to the audit

This is the single most important thing to understand when comparing this document with the audit, and the reason it is stated before the commands rather than after.

The audit's C-4 entry says:

Expected: SSH host keys and the TLS key present (they are baked — see build-auto-installer-iso.sh:345, :463-469), random-seed absent, machine-id absent or zero-length. Anything else changes F-03's severity.

That expectation described the broken state the audit found, and recording it was how the audit measured the size of F-03. Phase 10 plan 10-03 Task 2 removed that material. So:

After this change, the audit's stated expectation is the FAILURE condition. If SSH host keys or the TLS private key still appear in the tar, the strip layer did not run — most likely because a cached archipelago-rootfs.tar was reused. That is not a regression in the check; it is the check working.

The two negative findings the audit recorded are unchanged and must still hold: var/lib/systemd/random-seed absent, etc/machine-id absent or zero-length.

---

Commands to run

Run all of these on the build host, from the repo root, on a checkout that contains commits 21043096 and 408b328c.

1. Force a full rebuild

The strip layer lives inside the RECIPE_HASH region (between the # STEP 1: Build complete root filesystem and # STEP 2: Build minimal installer markers), so the hash changes and the cached tar is invalidated automatically. --rebuild is passed anyway so that a stale tar cannot mask the result for any reason:

UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild

UNBUNDLED=1 is mandatory per CLAUDE.md and project memory — the default env silently builds the wrong full-bundle variant.

2. List the identity artefacts in the shipped tar

WORK_DIR is image-recipe/build/auto-installer, so:

tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
  | grep -E 'etc/ssh/ssh_host|etc/machine-id|var/lib/systemd/random-seed|archipelago/ssl/archipelago'

3. Expected result after this plan

• no etc/ssh/ssh_host_* entries at all
• no etc/archipelago/ssl/archipelago.key and no archipelago.crt (the etc/archipelago/ssl/ directory must still be present — the first-boot staging swap needs somewhere to land)
• no var/lib/systemd/random-seed
• etc/machine-id present with size 0, or absent. Either satisfies "not shared"; record which one was actually observed rather than generalising.

Note on the TLS keypair specifically: it is now absent for two independent reasons, not one. The Dockerfile no longer generates it at all (that layer was removed so there is a single producer), and the strip layer still deletes it as belt-and-braces in case a future layer starts baking one. Seeing it present therefore means both defences were bypassed.

4. Confirm the provenance file rode along

tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar | grep rootfs-identity-stripped

Expected: one entry, opt/archipelago/rootfs-identity-stripped. Its absence means the strip layer did not execute and the whole check is void.

5. Confirm the regeneration path and its self-heal timer are still shipped

This is the brick check, and it is not optional. A stripped rootfs whose first-boot generation script failed to ship would leave every flashed node with no SSH host key and nothing to create one. The timer is part of the same check: without it, a node whose generators fail every in-boot retry has no unattended way back.

ls -l image-recipe/build/auto-installer/installer-iso/archipelago/scripts/first-boot-secrets.sh \
      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.service \
      image-recipe/build/auto-installer/installer-iso/archipelago/scripts/archipelago-first-boot-secrets.timer

Expected: all three present, first-boot-secrets.sh executable.

5b. Confirm the build-time generator assertion actually ran

The rootfs build fails outright if openssl or ssh-keygen is missing or non-executable, because that is the one way first-boot generation can fail deterministically — retries and reboots would never fix it, so it must never reach a node. A successful build therefore already proves the generators are present, and the build log says so:

grep 'first-boot secret generators present' <build log>

If you did not capture the log, assert it against the tar instead:

tar -tvf image-recipe/build/auto-installer/archipelago-rootfs.tar \
  | grep -E 'usr/bin/(openssl|ssh-keygen)$'

Expected: both present and mode -rwxr-xr-x.

6. Record the RECIPE_HASH the builder actually used

cat image-recipe/build/auto-installer/archipelago-rootfs.recipe.sha256

Caveat — do not compute this hash from the repo file. image-recipe/build-debian-iso.sh copies the archived builder to a temp path and rewrites its relative paths before exec'ing it, and RECIPE_HASH hashes "$0" — the rewritten copy. The hashed region contains 35 such rewritten path expressions, and SCRIPT_DIR is substituted with an absolute path, so the hash is specific to the build host and checkout location. For reference, hashing the region of the committed repo file directly gives d2dc4df5427fe73d48227aab08cdf6debfe8dd554e6b18e3718f8d37ea9d675c, which is expected to differ from the stamp above.

---

Result

Paste the raw output of steps 2, 4, 5 and 6 here, then set the status at the top of this document to VERIFIED with the date and build-host label.

(pending — not yet run on a build host)

Verdict: pending

---

What this does and does not prove

Proves (once run): the rootfs tar extracted verbatim onto every disk flashed from the ISO carries no SSH host key, no TLS private key and no populated machine-id — so a first-boot regeneration failure degrades to "no key, the service refuses to start" rather than "fleet-shared key, silently", which is the substance of F-03.

Does not prove: that two nodes flashed from the same ISO actually end up with different keys. That is audit item C-3 (§779) and needs two physical machines; it remains separately UNVERIFIED. C-4 is a build-host check only.

Guidance for C-3: SSH and TLS are now equally sharp signals

An earlier revision of this document said SSH host keys were the sharper divergence signal for C-3, because the installer had a per-install TLS fallback that would produce a differing cert even if first-boot generation had failed. That asymmetry no longer exists.

There is now exactly one producer of each secret — gen_tls() and gen_ssh() inside first-boot-secrets.sh — and no other code in the ISO build creates either. The Dockerfile no longer bakes a TLS keypair and the installer's "ensure SSL cert exists" block is gone. So for C-3, treat both the same way:

# on each node
ssh-keyscan -t ed25519 localhost 2>/dev/null | ssh-keygen -lf -
openssl x509 -in /etc/archipelago/ssl/archipelago.crt -noout -fingerprint -sha256

Pass: both fingerprints differ between the two nodes. Fail: either matches — a matching TLS fingerprint is now exactly as damning as a matching host key, whereas before it could have been explained away by the fallback.

Also check, on each node, that the run actually succeeded rather than merely being quiet:

ls -l /var/lib/archipelago/.secrets-regenerated          # present on a healthy node
cat /var/lib/archipelago/first-boot-secrets.failed 2>&1  # absent on a healthy node
systemctl status archipelago-first-boot-secrets.timer    # enabled; the self-heal path

The audit's original C-3 fail condition — a WARNING: line in the log alongside an existing marker — can no longer occur by construction: the marker is only written when both generators succeeded. If you ever see that combination, the fix has been reverted.