Archipelago — open-source initial import
This commit is contained in:
Archipelago
@@ -0,0 +1,551 @@
|
||||
---
|
||||
phase: 10-key-material-hardening
|
||||
plan: 03
|
||||
subsystem: iso-build
|
||||
tags: [security, iso, first-boot, systemd, ssh-host-keys, tls, machine-id, bash]
|
||||
|
||||
requires: []
|
||||
provides:
|
||||
- "Fail-closed first-boot per-device secret generation: the completion marker is written only when both TLS and SSH generation succeeded"
|
||||
- "A single producer per secret — gen_tls()/gen_ssh() are the only code in the ISO build that create the TLS keypair and SSH host keys"
|
||||
- "Retry-with-backoff inside a single boot, so a transient first-boot condition recovers without a reboot"
|
||||
- "archipelago-first-boot-secrets.timer — unattended self-heal every 15 minutes until generation succeeds"
|
||||
- "A build-time assertion that fails the ISO build if openssl or ssh-keygen is missing from the rootfs"
|
||||
- "Parse-back validation (openssl pkey / openssl x509) before the staging swap, so no service ever reads a truncated artefact"
|
||||
- "A durable failure record at /var/lib/archipelago/first-boot-secrets.failed plus console + journal + stderr on failure"
|
||||
- "FIRST_BOOT_SECRETS_ROOT / FIRST_BOOT_SECRETS_BACKOFF test seams on the generated first-boot script"
|
||||
- "An identity-free rootfs tar: no SSH host keys, no TLS keypair, machine-id truncated"
|
||||
- "/opt/archipelago/rootfs-identity-stripped build-time provenance marker"
|
||||
- "tests/first-boot-secrets/run-tests.sh — 6-case harness driving the shipped heredoc body against a temp root with stubbed generators"
|
||||
affects: [image-recipe, first-boot, sshd, nginx-tls]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Seam-for-testability over assertion-in-a-comment: FIRST_BOOT_SECRETS_ROOT prefixes every absolute path so the NEGATIVE property (on failure the marker is NOT created) can be forced and asserted. Same move the entropy fix in 8b51b7e2 made for the RNG."
|
||||
- "Test the shipped bytes, not a copy: the harness extracts the first-boot script from the builder heredoc between the SECRETSSCRIPT delimiters, so the test and the artefact cannot drift."
|
||||
- "Strip identity material in the last Dockerfile layer so fail-closed is structural (no key exists) rather than procedural (a script promises to replace it)."
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- tests/first-boot-secrets/run-tests.sh
|
||||
- docs/security/KEY-02-ROOTFS-EVIDENCE.md
|
||||
modified:
|
||||
- image-recipe/_archived/build-auto-installer-iso.sh
|
||||
|
||||
key-decisions:
|
||||
- "UNIFY, DO NOT DELETE. The defect in F-03 was never that a second attempt to create a key existed — it was that failure was silent and the marker lied about it. A second attempt is only dangerous when it is an unaudited second PRODUCER with its own idea of success, its own absent retry policy and its own absent failure record. So both secondary producers were folded out (the Dockerfile bake and the installer fallback) leaving one generator per secret, rather than 'keep the fallback' (leaves a silent second source) or 'delete the fallback and accept a dead node' (a false trade between security and UX)."
|
||||
- "Fail-closed governs SERVING; self-heal governs RECOVERING. These are separate properties and both must hold. Nothing serves on a key we did not generate; nothing dead-ends waiting for a human at a console."
|
||||
- "Self-heal uses a systemd timer, not a sleep loop in the script. A loop would hold a Type=oneshot open for hours and hide the failure from systemctl; the timer plus the service's existing ConditionPathExists=! costs a healthy node nothing and needs no teardown."
|
||||
- "The timer's enable uses a hand-written symlink fallback. Every other `chroot systemctl enable` here ends in `2>/dev/null || true`, which would silently drop the self-heal path — the one thing whose absence is invisible until a node is already broken."
|
||||
- "Consumers in `failed` state are explicitly restarted on success. try-reload-or-restart is a no-op on a failed unit, so without this a self-healed node would have valid keys on disk and nginx still down — recovery that isn't."
|
||||
- "Retry semantics: attempt count equals the number of FIRST_BOOT_SECRETS_BACKOFF entries, and the wait after the final attempt is skipped (a failed last attempt is terminal). With the default `2 8 20` that is 3 attempts at t=0s/2s/10s per generator; the trailing 20 is the ceiling if the list is lengthened. Documented in the script rather than left as a puzzle."
|
||||
- "`After=systemd-random-seed.service` added as its own unit line rather than appended to the existing After=local-fs.target, both because systemd accumulates After= lines and because the plan's acceptance criterion greps for exactly that string."
|
||||
- "/var/lib/dbus/machine-id is removed only when it is a real file, not when it is the symlink to /etc/machine-id that Debian normally ships. Deleting a live symlink risks a boot-time surprise for no gain; a real copy would be genuine shared state."
|
||||
|
||||
requirements-completed: []
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "A first-boot secret regeneration that fails does NOT set the completion marker, so the oneshot retries on the next boot (D-05)"
|
||||
requirement: KEY-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tests/first-boot-secrets/run-tests.sh#openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS"
|
||||
status: pass
|
||||
- kind: other
|
||||
ref: "Negative control: moving `touch \"$MARKER\"` back outside the success branch makes that case fail with MARKER-SET-ON-FAILURE (transcript below)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "Each generator is retried with backoff within a single boot before the boot is declared failed (D-05)"
|
||||
requirement: KEY-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tests/first-boot-secrets/run-tests.sh#ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "A terminal failure is loud: console + durable on-disk record + journal, not only a log file nobody reads (D-05)"
|
||||
requirement: KEY-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tests/first-boot-secrets/run-tests.sh#tls-fail case asserts first-boot-secrets.failed exists, names TLS, and stderr carries a FAILED line"
|
||||
status: pass
|
||||
- kind: other
|
||||
ref: "logger + tee -a $ROOT/dev/console emitted by shout(); the console leg cannot be exercised in a temp root and is UNVERIFIED on hardware"
|
||||
status: partial
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "The shipped rootfs tar contains no SSH host keys, no TLS private key and no populated machine-id"
|
||||
requirement: KEY-04
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "docs/security/KEY-02-ROOTFS-EVIDENCE.md — requires an ISO build host; commands recorded, not yet run"
|
||||
status: blocked
|
||||
human_judgment: true
|
||||
- id: D5
|
||||
description: "The regeneration script is exercised by an automated test that fails when the marker is set on a failed run"
|
||||
requirement: KEY-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tests/first-boot-secrets/run-tests.sh — 6/6 PASS; negative control reproduced"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D6
|
||||
description: "Exactly one producer per secret: no second code path anywhere in the ISO build can mint a TLS key or SSH host key with its own accounting"
|
||||
requirement: KEY-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tests/first-boot-secrets/run-tests.sh#single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh"
|
||||
status: pass
|
||||
- kind: other
|
||||
ref: "Negative control: reintroducing the installer's chroot openssl req block turns case 6 red naming the line, and nothing else"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D7
|
||||
description: "A failure self-heals unattended — it never dead-ends a node whose only exit is physical access"
|
||||
requirement: KEY-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tests/first-boot-secrets/run-tests.sh#self-heal: failed run then a later successful run -> key present, marker set, failed units restarted"
|
||||
status: pass
|
||||
- kind: other
|
||||
ref: "archipelago-first-boot-secrets.timer installed + enabled (symlink fallback) — UNVERIFIED on hardware; the harness proves the script half, not systemd's scheduling"
|
||||
status: partial
|
||||
human_judgment: false
|
||||
- id: D8
|
||||
description: "The deterministic total-failure cause (missing generator binary) fails the BUILD, not the fleet"
|
||||
requirement: KEY-04
|
||||
verification:
|
||||
- kind: other
|
||||
ref: "Rootfs RUN assertion on /usr/bin/openssl and /usr/bin/ssh-keygen; fires during the container build. UNVERIFIED until a build host runs it — see KEY-02-ROOTFS-EVIDENCE.md step 5b"
|
||||
status: blocked
|
||||
human_judgment: true
|
||||
|
||||
duration: 1h
|
||||
completed: 2026-08-02
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 10 Plan 03: Fail-closed first-boot secrets + identity-free rootfs — Summary
|
||||
|
||||
First-boot per-device secret regeneration now retries with backoff and then fails closed, and
|
||||
the rootfs tar it repairs no longer carries the fleet-shared SSH host keys, TLS keypair or
|
||||
machine-id it was silently papering over. Closes the build side of audit finding **F-03**.
|
||||
|
||||
> **Task 3 (C-4 build-host evidence) is a blocking checkpoint and is NOT done.** It needs an
|
||||
> ISO build host. `docs/security/KEY-02-ROOTFS-EVIDENCE.md` carries the exact command sequence
|
||||
> and is marked UNVERIFIED. Nothing in this plan claims the tar listing was observed.
|
||||
|
||||
## What was wrong
|
||||
|
||||
`first-boot-secrets.sh` (a heredoc inside `image-recipe/_archived/build-auto-installer-iso.sh`,
|
||||
which is **live** — `image-recipe/build-debian-iso.sh` execs it) had two fail-open branches that
|
||||
logged `WARNING: ... keeping baked key` and continued, and `touch "$MARKER"` ran unconditionally
|
||||
**outside both `if` blocks**. Combined with the unit's
|
||||
`ConditionPathExists=!/var/lib/archipelago/.secrets-regenerated` and the script's own
|
||||
`[ -f "$MARKER" ] && exit 0`, one transient failure at first boot left that node on the
|
||||
image-wide shared SSH host key and TLS private key **permanently and silently** — and the ISO is
|
||||
a published artefact, so every downloader holds those keys.
|
||||
|
||||
## Commits
|
||||
|
||||
| Commit | Task | What |
|
||||
|---|---|---|
|
||||
| `21043096` | 1 | Fail-closed, retried regeneration + `tests/first-boot-secrets/run-tests.sh` |
|
||||
| `408b328c` | 2 | Rootfs identity-strip layer + two comment corrections that follow from it |
|
||||
| `201ef474` | 3 (prep) | `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, marked UNVERIFIED |
|
||||
| `2efab5f2` | follow-up | Single-producer unification, build-time generator assertion, self-heal timer, 3 new test cases |
|
||||
| `d9b3a7d5` | follow-up | Quote the `Dockerfile.rootfs` heredoc so comments cannot execute (closes deferred D1) |
|
||||
| `40b77e39` | follow-up | Refuse to bless a cert minted under an untrustworthy clock; backdate `notBefore` |
|
||||
|
||||
Nothing was pushed, tagged, built or deployed, per the execution brief.
|
||||
|
||||
## Task 1 — fail-closed regeneration
|
||||
|
||||
- `ROOT="${FIRST_BOOT_SECRETS_ROOT:-}"` prefixes every absolute path. Unset in production the
|
||||
expansion is empty and behaviour is byte-identical; set, it is what makes the negative
|
||||
property assertable at all.
|
||||
- `retry()` runs each generator up to N times with waits from `FIRST_BOOT_SECRETS_BACKOFF`
|
||||
(default `2 8 20`). Staging-then-swap is preserved for both generators, with `.new` files
|
||||
removed on failure so no half-keypair is left behind.
|
||||
- `touch "$MARKER"` now lives inside a `TLS_OK == 1 && SSH_OK == 1` branch. Any other outcome
|
||||
writes `/var/lib/archipelago/first-boot-secrets.failed` (timestamp, which generator failed,
|
||||
both flags), shouts to console + `logger` + stderr, and `exit 1` so the unit lands in `failed`
|
||||
rather than `active`. A later successful boot deletes the record so a recovered node does not
|
||||
carry a stale alarm.
|
||||
- `After=systemd-random-seed.service` added to the unit. A no-op today (no seed file is baked,
|
||||
which the audit verified) and correct if one is ever introduced.
|
||||
- The script header states the operational trade in plain words, including that recovery from a
|
||||
terminal failure needs the physical console.
|
||||
|
||||
### Harness results (final, all six cases)
|
||||
|
||||
```
|
||||
extracted 236 lines from the builder; bash -n clean
|
||||
PASS: both generators succeed -> exit 0, marker set, keys swapped in
|
||||
PASS: openssl fails every attempt -> exit non-zero, NO marker, failure record names TLS
|
||||
PASS: ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)
|
||||
PASS: TLS fails every attempt on a stripped root -> NO key, NO marker, non-zero exit, record names TLS
|
||||
PASS: self-heal: failed run then a later successful run -> key present, marker set, failed units restarted
|
||||
PASS: single-producer invariant: every key-creating invocation is inside gen_tls/gen_ssh
|
||||
|
||||
──────── first-boot-secrets summary ────────
|
||||
passed: 6 failed: 0
|
||||
```
|
||||
|
||||
### Negative control (required by the plan's acceptance criteria)
|
||||
|
||||
`touch "$MARKER"` moved back outside the success branch, harness re-run, then reverted:
|
||||
|
||||
```
|
||||
SCRATCH APPLIED: marker touch moved back outside the success branch
|
||||
--- harness against the fail-open variant ---
|
||||
extracted 175 lines from the builder; bash -n clean
|
||||
PASS: both generators succeed -> exit 0, marker set, keys swapped in
|
||||
FAIL: openssl fails every attempt -> MARKER-SET-ON-FAILURE
|
||||
exit=1 root=/tmp/tmp.Ta1YFhHWdi/root-tls-fail
|
||||
stderr: ARCHIPELAGO FIRST BOOT FAILED: could not generate this device's TLS key material. ...
|
||||
PASS: ssh-keygen fails twice then succeeds -> backoff recovers within one boot (3 attempts)
|
||||
|
||||
──────── first-boot-secrets summary ────────
|
||||
passed: 2 failed: 1
|
||||
EXIT=1
|
||||
```
|
||||
|
||||
The test fails on exactly the regression it exists to pin, and only that case.
|
||||
|
||||
## Task 2 — identity-free rootfs
|
||||
|
||||
Final `RUN` layer added to `Dockerfile.rootfs`, after every package install and after the
|
||||
`openssl req` layer, so nothing regenerates the material afterwards:
|
||||
|
||||
- `rm -f /etc/ssh/ssh_host_*` (private keys and `.pub` alike)
|
||||
- `rm -f` the archipelago TLS key and crt, keeping the `/etc/archipelago/ssl` directory
|
||||
- `: > /etc/machine-id` (systemd's documented regenerate-on-next-boot state)
|
||||
- `/var/lib/dbus/machine-id` removed only if it is a real file, not the usual symlink
|
||||
- writes `/opt/archipelago/rootfs-identity-stripped` listing what it removed, with **no**
|
||||
timestamp so RECIPE_HASH reproducibility is unaffected
|
||||
|
||||
The `openssl req` layer is deliberately unmodified.
|
||||
|
||||
**RECIPE_HASH changed.** The strip layer is inside the hashed region
|
||||
(`sed -n '/^# STEP 1.../,/^# STEP 2.../p' | grep -c rootfs-identity-stripped` → 1), so the next
|
||||
build is forced to rebuild the rootfs tar. Task 3's evidence would be meaningless against a
|
||||
cached tar, and `--rebuild` is specified as well.
|
||||
|
||||
`grep -c 'ssh_host'` on the builder went **3 → 6**.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### 1. [Rule 1 — Bug] Backticks in my own strip-layer comment would have hung every ISO build
|
||||
|
||||
- **Found during:** Task 2, self-check of the added block.
|
||||
- **Issue:** `Dockerfile.rootfs` is written with an **unquoted** heredoc (`<<DOCKERFILE`), so
|
||||
backticks in its body are command substitution evaluated by the build shell. Two comment
|
||||
lines I wrote contained `` `openssl req` ``. Reproduced in isolation: the heredoc hung for the
|
||||
full 2-minute timeout as `openssl req` waited on stdin. `bash -n` is clean on this — syntax
|
||||
checking cannot catch it.
|
||||
- **Fix:** replaced with double quotes, and added an explicit `NOTE:` in the block warning that
|
||||
the heredoc is unquoted and backticks must never appear there.
|
||||
- **Commit:** `408b328c`
|
||||
|
||||
### 2. [Rule 1 — Correctness] Script header claim about TLS, twice corrected
|
||||
|
||||
- **Found during:** Task 2, after discovering the installer's TLS fallback.
|
||||
- **Issue:** the Task 1 header claimed "the nginx TLS listener will not start". With the
|
||||
installer fallback in place that was false — the web UI would still come up. Shipping a
|
||||
confident false statement in a security-critical script is worse than shipping none.
|
||||
- **First fix (`408b328c`):** narrowed the claim to SSH only, and described the TLS fallback
|
||||
honestly as per-install, never image-wide.
|
||||
- **Second fix (`2efab5f2`):** the fallback is gone, so the original claim is true again for
|
||||
both. Restored, with the reasoning attached rather than left implicit. No comment anywhere in
|
||||
the builder now implies a TLS fallback exists.
|
||||
|
||||
### 3. [Rule 2 — Threat coverage] `/var/lib/dbus/machine-id`
|
||||
|
||||
- **Issue:** T-10-26 is machine-id correlation across nodes. The plan named `/etc/machine-id`
|
||||
only. If dbus ships a real copy rather than the usual symlink, truncating `/etc/machine-id`
|
||||
alone leaves correlated state.
|
||||
- **Fix:** guarded removal — symlinks are left alone, real files are removed.
|
||||
- **Commit:** `408b328c`
|
||||
|
||||
## Follow-up: unify to a single producer (`2efab5f2`)
|
||||
|
||||
The installer's TLS fallback prompted a decision cycle worth recording, because the reasoning
|
||||
matters more than the outcome.
|
||||
|
||||
**The false trade.** The question was framed as "keep the fallback (a second source of keys) or
|
||||
delete it (a first-boot failure costs the user the web UI, recoverable only at the console)".
|
||||
Both options were wrong, and the framing was wrong. **The defect in F-03 was never that a second
|
||||
attempt to create a key existed. It was that failure was silent and the completion marker lied
|
||||
about it.** A second attempt is only dangerous when it is an unaudited second *producer* —
|
||||
carrying its own idea of success, its own absent retry policy, its own absent failure record.
|
||||
So the fix is to unify, not to delete and accept a dead node.
|
||||
|
||||
**What shipped:**
|
||||
|
||||
1. **One producer per secret.** `gen_tls()` and `gen_ssh()` are the only code in the ISO build
|
||||
that create the TLS keypair and the SSH host keys. Two secondary producers were folded out:
|
||||
the Dockerfile's `openssl req` layer (which baked a keypair the strip layer deleted moments
|
||||
later in the same build) and the installer's "ensure SSL cert exists" block. The invariant is
|
||||
checked mechanically, not asserted in prose — case 6 of the harness fails if any executable
|
||||
`openssl req` / `ssh-keygen -A` invocation appears outside the generator heredoc.
|
||||
|
||||
2. **The deterministic failure is caught at build time.** The one realistic way generation fails
|
||||
on every retry forever is a missing generator binary, and that is deterministic — no retry or
|
||||
reboot fixes it. A rootfs `RUN` layer now fails the build if `/usr/bin/openssl` or
|
||||
`/usr/bin/ssh-keygen` is missing or non-executable. **The build already guaranteed these**
|
||||
(`openssl` and `openssh-server` are both in the package list, and `openssh-server`
|
||||
hard-depends `openssh-client`, which ships `ssh-keygen`), so this is cheap insurance rather
|
||||
than a fix. It earns its place the first time someone edits that package list.
|
||||
|
||||
3. **Failure self-heals; it never dead-ends.** `archipelago-first-boot-secrets.timer`
|
||||
(`OnBootSec=5min`, `OnUnitActiveSec=15min`) re-runs the service until it succeeds. The
|
||||
service's existing `ConditionPathExists=!` makes every trigger a no-op once the marker
|
||||
exists, so a healthy node pays nothing and no teardown is needed. Two details that would have
|
||||
made this theatre if missed:
|
||||
- `chroot systemctl enable` can fail silently, and every other enable in this file ends in
|
||||
`|| true`. The timer's enable has a hand-written symlink fallback, because the absence of
|
||||
self-heal is invisible until a node is already broken.
|
||||
- `try-reload-or-restart` is a **no-op on a failed unit**. Without special handling, a
|
||||
self-healed node would have valid keys on disk and nginx still down. Consumers found in
|
||||
`failed` are now explicitly restarted (`--no-block`, to avoid a boot-transaction deadlock
|
||||
at first boot, where we are ordered `Before=` them).
|
||||
|
||||
4. **Never serve a bogus key.** `gen_tls` now parses both halves back (`openssl pkey`,
|
||||
`openssl x509`) before the staging swap, so a truncated or half-written artefact is never
|
||||
what nginx reads. Fail-closed governs *serving*; retry-and-self-heal governs *recovering*.
|
||||
They are different properties and both hold.
|
||||
|
||||
### Negative controls for the three new cases
|
||||
|
||||
Each defect was reintroduced, the suite run, and the defect reverted. Each lights up **exactly
|
||||
one** case — a test that goes red for several reasons at once is not pinning any of them.
|
||||
|
||||
**Control A — reintroduce a fallback-style key creation on the failure path** (the deleted
|
||||
installer block's behaviour, moved into the script):
|
||||
|
||||
```
|
||||
FAIL: TLS fails every attempt on a stripped root -> TLS-KEY-EXISTS-AFTER-FAILURE TLS-CRT-EXISTS-AFTER-FAILURE
|
||||
passed: 5 failed: 1
|
||||
```
|
||||
|
||||
*(First run of this control also reddened case 5, because case 5's run-1 block redundantly
|
||||
re-asserted case 4's property. That assertion was removed — case 5 now tests recovery only —
|
||||
and the control re-run to confirm it is isolated. The transcript above is the re-run.)*
|
||||
|
||||
**Control B — dead-end a node that has already failed once** (`exit 0` early if the failure
|
||||
record exists, a plausible "don't retry a known-bad node" optimisation):
|
||||
|
||||
```
|
||||
FAIL: self-heal -> run2-marker-missing run2-key-missing run2-crt-missing run2-stale-failure-record run2-did-not-restart-failed-nginx
|
||||
passed: 5 failed: 1
|
||||
```
|
||||
|
||||
**Control C — reintroduce the installer's `chroot ... openssl req` block verbatim:**
|
||||
|
||||
```
|
||||
FAIL: single-producer invariant -> SECOND-PRODUCER-at-line-3586
|
||||
generator heredoc spans lines 1713-1950 of image-recipe/_archived/build-auto-installer-iso.sh
|
||||
passed: 5 failed: 1
|
||||
```
|
||||
|
||||
All three reverted; suite back to 6/6.
|
||||
|
||||
## Residual operational risk — stated plainly
|
||||
|
||||
**A machine on which secret generation can never succeed ends up with no SSH host key and no
|
||||
TLS key. sshd will not start, nginx will not serve the web UI, and that node needs physical
|
||||
console access.** That is the honest worst case and it is not softened anywhere in the code
|
||||
comments either.
|
||||
|
||||
What shrinks it to genuinely-broken-hardware:
|
||||
|
||||
- **The deterministic cause is gone before shipping.** A missing `openssl` or `ssh-keygen` fails
|
||||
the ISO build, so it cannot reach a node.
|
||||
- **Transient causes are absorbed.** Three attempts with backoff inside the boot (proven by
|
||||
harness case 3, which shows a generator failing twice and succeeding on the third), then every
|
||||
15 minutes on the timer, then again on every boot — indefinitely, because the marker is never
|
||||
written on failure.
|
||||
- **Recovery completes itself.** On a later success the script restarts the units that refused
|
||||
to start, so the node comes back without a reboot and without a human (harness case 5).
|
||||
|
||||
What is left is a machine where `openssl` or `ssh-keygen` is present but cannot ever produce a
|
||||
key — a disk that is permanently full, or failing hardware. On that machine the node refuses to
|
||||
serve rather than serving on a key nobody generated, which is the trade this phase exists to
|
||||
make. It says so on the console, in the journal, and in
|
||||
`/var/lib/archipelago/first-boot-secrets.failed`.
|
||||
|
||||
## Follow-up: quote the Dockerfile heredoc (`d9b3a7d5`) — closes deferred D1
|
||||
|
||||
`cat > "$WORK_DIR/Dockerfile.rootfs" <<DOCKERFILE` was **unquoted**, so the build shell
|
||||
performed command substitution on the Dockerfile body: a backtick inside a Dockerfile *comment*
|
||||
was executed on the build host and its output spliced into the generated file. Six comments did
|
||||
this, and one of them ran `systemctl start archipelago-fips.service` against the build machine
|
||||
on every ISO build.
|
||||
|
||||
**Boundary checked before editing.** Only lines inside the heredoc body are at risk. The other
|
||||
backticked comments in this file (`:264`, `:809`, `:1188`, `:1289`, `:1506`, `:1605`, `:3597`,
|
||||
`:3651`) are ordinary shell comments outside any unquoted heredoc, plus one inside the *quoted*
|
||||
`SECRETSSCRIPT` heredoc — none were ever evaluated, and none were touched.
|
||||
|
||||
**Fixed the class, not the instances.** The body needs exactly four build-time values, all
|
||||
package names (`LINUX_IMAGE_PKG`, `GRUB_EFI_PKG`, `GRUB_EFI_SIGNED_PKG`, `GRUB_PC_PKG`), on four
|
||||
consecutive lines — so quoting was entirely practical. The heredoc is split into
|
||||
`DOCKERFILE_HEAD` and `DOCKERFILE_TAIL`, both quoted, with one explicit `printf` interpolating
|
||||
those four names between them. Escapes that existed *only* because the heredoc was unquoted were
|
||||
undone in the same pass: six trailing `\\` → `\` (Docker line continuations) and four `\$` → `$`
|
||||
(RUN arguments reach the shell verbatim; Docker does not substitute variables in RUN).
|
||||
|
||||
**Substance verified by rendering, not by inspection.** The generated Dockerfile was rendered
|
||||
before and after with identical inputs and diffed *normalised* (continuations joined, whitespace
|
||||
collapsed). Both are 190 normalised lines and the only differences are the six comments regaining
|
||||
their text — every instruction byte-identical:
|
||||
|
||||
```
|
||||
< # the archipelago backend calls
|
||||
> # the archipelago backend calls `systemctl start archipelago-fips.service`
|
||||
< # fips-gateway is gated behind the Cargo feature (depends on
|
||||
> # fips-gateway is gated behind the `gateway` Cargo feature (depends on
|
||||
```
|
||||
|
||||
**Case 7** asserts every heredoc writing `Dockerfile.rootfs` has a quoted delimiter, and when one
|
||||
does not, reports which body lines would execute. The assertion is on the **delimiter, not on
|
||||
backticks** — with quoting a backticked comment is legal, and six of them are back in the body on
|
||||
purpose; flagging backticks would flag a non-bug and fail on the very comments this restored.
|
||||
|
||||
Controls:
|
||||
|
||||
```
|
||||
Control D — unquote the delimiter (the real regression):
|
||||
FAIL: Dockerfile heredoc quoting -> UNQUOTED-DELIMITER-at-line-287 would-execute-at-lines:317,318
|
||||
passed: 6 failed: 1
|
||||
|
||||
Control E — add a backticked comment, delimiter still quoted:
|
||||
PASS: Dockerfile heredoc delimiters are quoted — a backticked comment cannot execute
|
||||
passed: 7 failed: 0
|
||||
and it renders intact:
|
||||
175:# Control E: a backticked `systemctl start archipelago-fips.service` comment
|
||||
```
|
||||
|
||||
Control E is the more informative of the two: the backtick that used to be a build-host RCE is
|
||||
now inert and renders as written. That is what "fixed the class" means, and it is why a bare
|
||||
backtick reintroduction correctly reddens nothing.
|
||||
|
||||
`deferred-items.md` held D1 as its only entry and has been **deleted** — nothing was left that is
|
||||
genuinely out of scope.
|
||||
|
||||
## Follow-up: untrustworthy clock at cert-minting time (`40b77e39`)
|
||||
|
||||
The failure fail-closed cannot catch, **because generation succeeds**. This unit runs before time
|
||||
has synced; `openssl req -x509` stamps `notBefore` from whatever the clock says. Dead RTC or flat
|
||||
CMOS battery → clock ahead gives "not yet valid" (harder to diagnose than a self-signed warning),
|
||||
clock behind gives an already-expired cert once time syncs. The marker was then set and never
|
||||
revisited: a node permanently serving a cert nothing accepts.
|
||||
|
||||
**Finding, established rather than assumed:** this image does **not** use `systemd-timesyncd`. It
|
||||
installs and enables **chrony** (`:388`, `:575`), and `chrony-wait.service` — the unit that is
|
||||
`Before=time-sync.target` — is **not** enabled. So `time-sync.target` is inert here and ordering
|
||||
`After=` it would buy nothing. Enabling `chrony-wait` to make it meaningful would stall boot
|
||||
behind NTP on a node with no network, and these nodes are routinely offline at first boot.
|
||||
|
||||
**Decision: no ordering change.** Not deadlocking boot outranks cert-date elegance (constraint 3).
|
||||
Fixed locally instead:
|
||||
|
||||
1. **Backdate `notBefore` by 24h** so ordinary node/client skew cannot invalidate a fresh cert.
|
||||
This does not weaken a self-signed cert — `notBefore` is not a security control here.
|
||||
`-not_before`/`-not_after` arrived in OpenSSL 3.5 and the rootfs is `debian:trixie` which
|
||||
ships it, but the capability is **probed, not assumed**: guessing wrong would fail every
|
||||
attempt and brick the node, the exact outcome all of this exists to prevent. Without the flags
|
||||
we do not backdate, and rule 2 still covers the dangerous case.
|
||||
2. **Refuse to bless a cert dated outside a plausible window** (2026-01-01 … 2056-01-01). The
|
||||
material stays installed — the node is usable, sshd comes up — but the dates are recorded as
|
||||
`failed=cert-dates` and the cert is regenerated automatically once time syncs.
|
||||
|
||||
Generation is now driven by **need** rather than "is the marker absent", and
|
||||
`ConditionPathExists=!` was removed from the unit so a node that already completed can still be
|
||||
re-examined. Skipping the unit is precisely how such a node would stay broken forever. On a
|
||||
healthy node the script exits in milliseconds.
|
||||
|
||||
**Anti-spin is one condition:** a date-driven regeneration happens *only* when the clock is
|
||||
currently plausible. A node whose clock is still wrong re-checks and mints nothing.
|
||||
|
||||
### Regression I introduced and caught
|
||||
|
||||
Driving generation purely by content made `needs_ssh()` false whenever *any* host key existed —
|
||||
which would have left an image-baked, fleet-shared host key in place forever. **That is F-03
|
||||
reopened.** The marker check is back in both `needs_` functions. Case 1 — which prestages a baked
|
||||
key and asserts it was replaced — is what caught it.
|
||||
|
||||
### Controls
|
||||
|
||||
```
|
||||
Control F — bless the cert regardless of clock (the pre-fix behaviour):
|
||||
FAIL: wrong clock -> run1-BAD-DATES-NOT-RECORDED
|
||||
passed: 7 failed: 1
|
||||
|
||||
Control G — remove the anti-spin guard:
|
||||
FAIL: wrong clock -> SPINNING-reminted-while-clock-still-wrong(1->2)
|
||||
passed: 7 failed: 1
|
||||
```
|
||||
|
||||
**Control G first passed against a deliberately broken guard**, which was a flaw in my test, not
|
||||
in the fix: the assertion compared certificate dates, and a re-mint under a frozen fake clock
|
||||
produces a byte-identical `notBefore`. Dates cannot distinguish "left alone" from "regenerated
|
||||
again". The assertion now counts `openssl req` invocations, which can — and only then did the
|
||||
control redden. Worth recording as the second time in this plan that a first-draft assertion
|
||||
looked green for the wrong reason.
|
||||
|
||||
### Not covered here
|
||||
|
||||
Nodes already deployed from earlier ISOs **never receive this script** — it is installed by the
|
||||
installer, not shipped by OTA. Fleet remediation for those nodes is 10-04/OTA work in `core/**`,
|
||||
which is held by other executors, so per the standing constraint it is reported rather than
|
||||
attempted.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None. No placeholder values, no TODOs, no unwired code paths.
|
||||
|
||||
## Threat Flags
|
||||
|
||||
None. No new network endpoint, auth path, file-access pattern or schema change at a trust
|
||||
boundary. The plan installs no packages (T-10-SC: accept); none were added.
|
||||
|
||||
## UNVERIFIED — needs hardware
|
||||
|
||||
Task 3's C-4 checkpoint is now **more** important, not less: with the rootfs stripped and no
|
||||
install-time fallback, the tar listing is the only pre-hardware evidence that the shipped image
|
||||
is identity-free.
|
||||
|
||||
| Item | Audit ref | What it needs | Command |
|
||||
|---|---|---|---|
|
||||
| Rootfs tar is identity-free after a forced rebuild | **C-4** | ISO build host with podman/docker and disk for a full rootfs rebuild | `UNBUNDLED=1 bash image-recipe/build-debian-iso.sh --rebuild`, then the tar listings in `docs/security/KEY-02-ROOTFS-EVIDENCE.md` steps 2/4/5/5b/6 |
|
||||
| The build-time generator assertion actually fires | **C-4** | same build host | `grep 'first-boot secret generators present' <build log>` — evidence doc step 5b |
|
||||
| The self-heal timer ships and is enabled on the target | — | same build host, then a node | evidence doc step 5 (timer present on installer media); `systemctl status archipelago-first-boot-secrets.timer` on a node |
|
||||
| Two nodes flashed from one ISO get different keys | **C-3** | two physical machines | audit §779; both SSH and TLS fingerprints are now equally sharp signals — see the C-3 section of the evidence doc |
|
||||
| The console leg of the failure shout reaches a real screen | — | a real node, or a VM console | force a first-boot failure and observe `/dev/console` |
|
||||
|
||||
The harness proves the *script* half of self-heal (a failed run followed by a successful run
|
||||
recovers the node and restarts the failed units). It does not and cannot prove systemd's
|
||||
scheduling — that the timer is enabled and actually fires at 5min/15min. That is hardware
|
||||
verification.
|
||||
|
||||
## Self-Check
|
||||
|
||||
- `image-recipe/_archived/build-auto-installer-iso.sh` — FOUND, `bash -n` clean
|
||||
- `tests/first-boot-secrets/run-tests.sh` — FOUND, mode 755, exits 0 with 8 PASS
|
||||
- `docs/security/KEY-02-ROOTFS-EVIDENCE.md` — FOUND, contains `C-4`
|
||||
- `deferred-items.md` — DELETED; its only entry (D1) is fixed, not filed
|
||||
- Commits `21043096`, `408b328c`, `201ef474`, `2efab5f2`, `d9b3a7d5`, `40b77e39` — all FOUND
|
||||
- Generated `Dockerfile.rootfs` rendered before/after the heredoc change and diffed normalised:
|
||||
190 lines each, only the six comment restorations differ
|
||||
- Single-producer grep: the only executable key-creating invocations in the builder are
|
||||
`openssl req` and `ssh-keygen -A` inside the generator heredoc; every other match is a comment
|
||||
- `git status --porcelain image-recipe/` — clean; `_archived/` not moved or renamed
|
||||
- No file authored by a concurrent agent (`core/archipelago/src/**`, `neode-ui/**`,
|
||||
`.planning/STATE.md`) was staged in any commit
|
||||
|
||||
## Self-Check: PASSED
|
||||
Reference in New Issue
Block a user