lfg2025/archy
1
0
Fork 0
Code Issues 25 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

fix(10-03): don't bless a cert minted under an untrustworthy clock

Browse Source 
The failure fail-closed cannot catch, because generation SUCCEEDS.

This unit runs very early (DefaultDependencies=no, Before=ssh/nginx), long
before time has synced. `openssl req -x509` stamps notBefore from whatever the
clock says, so on a node with a dead RTC or a flat CMOS battery the cert can be
years out: clock ahead -> clients reject it as "not yet valid", a harder failure
than the usual self-signed warning; clock behind -> notAfter is already in the
past once time syncs. The completion marker was then set and never revisited —
a node permanently serving a cert nothing accepts.

Finding 1, reported rather than assumed: this image does NOT use
systemd-timesyncd. It installs and enables chrony, and chrony-wait.service —
the unit that is Before=time-sync.target — is not enabled. So time-sync.target
is inert here and ordering After= it would buy nothing. Enabling chrony-wait to
make it meaningful would stall boot behind NTP on a node with no network, and
these nodes are routinely offline at first boot. Not deadlocking boot outranks
cert-date elegance, so the ordering is deliberately left alone.

Fixed locally instead, in two parts:

1. Backdate notBefore by 24h so ordinary skew between node and client cannot
   invalidate a fresh cert. -not_before/-not_after arrived in OpenSSL 3.5 and
   the rootfs is debian:trixie which ships it, but the capability is PROBED,
   not assumed — guessing wrong would fail every attempt and brick the node,
   the exact outcome all of this exists to prevent. Without the flags we simply
   do not backdate and rule 2 still covers the dangerous case.

2. Refuse to bless a cert dated by a clock outside a plausible window
   (2026-01-01 .. 2056-01-01). The material stays installed so the node is
   usable and sshd comes up, but the bad dates are recorded as
   failed=cert-dates and the cert is regenerated automatically once time syncs.

Generation is now driven by need rather than by "is the marker absent", and
ConditionPathExists=! is removed from the unit so a node that already completed
can still be re-examined — skipping the unit is precisely how such a node stays
broken forever. The script exits in milliseconds when everything is fine.

Anti-spin is one condition: a date-driven regeneration happens ONLY when the
clock is currently plausible. A node whose clock is still wrong re-checks and
mints nothing.

Regression caught while writing this: driving generation purely by content made
needs_ssh() false whenever any host key existed, which would have left an
image-baked fleet-shared key in place forever — F-03 reopened. The marker check
is back in both needs_ functions and case 1 (which prestages a baked key and
asserts it was replaced) is what caught it.

Case 8 covers mint-under-wrong-clock, repair-after-sync, and both spin
directions. Controls: blessing regardless of clock reddens only case 8
(run1-BAD-DATES-NOT-RECORDED); removing the anti-spin guard reddens only case 8
(SPINNING-reminted-while-clock-still-wrong(1->2)). The second control initially
passed against a broken guard because the assertion compared certificate dates,
and a re-mint under a frozen clock produces a byte-identical notBefore — the
assertion now counts mints, which is the only thing that distinguishes "left
alone" from "regenerated again".

Not covered here: nodes already deployed from earlier ISOs never receive this
script (it is installed by the installer, not by OTA), so fleet remediation for
them remains 10-04/OTA work in core/**, which is held by other executors.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-02 10:08:17 -04:00
co-authored by Claude Opus 5
parent 9622926868
commit 40b77e392a
2 changed files with 315 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/first-boot-secrets/run-tests.sh
+120 -8
View File
@@ -81,10 +81,15 @@ make_stubs() {
# non-empty checks are exercised for real, and implements the `pkey`/`x509`
# parse-back validation the generator does before it swaps.
sub="${1:-}"
# Capability probe. The script asks `openssl req -help` whether it can backdate.
if [ "$sub" = "req" ] && [ "${2:-}" = "-help" ]; then
[ "${STUB_OPENSSL_NOT_BEFORE:-yes}" = "yes" ] && echo " -not_before val stub"
exit 0
fi
case "$sub" in
pkey|x509)
# Validation: succeed iff the file exists and is non-empty. This is what
# lets the harness prove a truncated artefact is never swapped in.
pkey)
f=""
while [ $# -gt 0 ]; do
case "$1" in
@@ -95,19 +100,58 @@ case "$sub" in
[ -n "$f" ] && [ -s "$f" ] || exit 1
exit 0
;;
x509)
# Validation (-noout -in f) plus date readback. The stub cert carries
# the epochs it was minted with, so the harness can drive the script's
# date arithmetic without a real certificate.
f=""; want_start=0; want_end=0
while [ $# -gt 0 ]; do
case "$1" in
-in) f="$2"; shift 2 ;;
-startdate) want_start=1; shift ;;
-enddate) want_end=1; shift ;;
*) shift ;;
esac
done
[ -n "$f" ] && [ -s "$f" ] || exit 1
if [ "$want_start" = 1 ] || [ "$want_end" = 1 ]; then
nb=$(sed -n 's/^STUB_NOTBEFORE=//p' "$f"); na=$(sed -n 's/^STUB_NOTAFTER=//p' "$f")
[ -n "$nb" ] && [ -n "$na" ] || exit 1
[ "$want_start" = 1 ] && echo "notBefore=$(date -u -d "@$nb" '+%b %e %H:%M:%S %Y GMT')"
[ "$want_end" = 1 ] && echo "notAfter=$(date -u -d "@$na" '+%b %e %H:%M:%S %Y GMT')"
fi
exit 0
;;
esac
# Count real mints. Comparing certificate dates cannot detect a re-mint when
# the clock is frozen — the second cert carries the same notBefore — so the
# anti-spin assertions count invocations instead.
reqcount="${STUB_COUNTER_DIR:-/tmp}/openssl-req.count"
rn=$(cat "$reqcount" 2>/dev/null || echo 0)
echo $((rn + 1)) > "$reqcount"
[ "${STUB_OPENSSL_MODE:-ok}" = "fail" ] && exit 1
keyout="" out=""
keyout="" out="" nb="" na="" days=""
while [ $# -gt 0 ]; do
case "$1" in
-keyout) keyout="$2"; shift 2 ;;
-out) out="$2"; shift 2 ;;
*) shift ;;
-keyout) keyout="$2"; shift 2 ;;
-out) out="$2"; shift 2 ;;
-not_before) nb="$2"; shift 2 ;;
-not_after) na="$2"; shift 2 ;;
-days) days="$2"; shift 2 ;;
*) shift ;;
esac
done
# Mirror openssl: -not_before/-not_after win; otherwise notBefore is "now" and
# notAfter is now + days. "now" honours the harness's fake clock.
now="${FIRST_BOOT_SECRETS_NOW:-$(date -u +%s)}"
# YYYYMMDDHHMMSSZ -> something GNU date can parse
asn1() { printf '%s' "$1" | sed -E 's/^([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})Z?$/\1-\2-\3 \4:\5:\6 UTC/'; }
if [ -n "$nb" ]; then nb_epoch=$(date -u -d "$(asn1 "$nb")" +%s 2>/dev/null || echo "$now"); else nb_epoch="$now"; fi
if [ -n "$na" ]; then na_epoch=$(date -u -d "$(asn1 "$na")" +%s 2>/dev/null || echo $((now + 315360000))); else na_epoch=$((now + ${days:-3650} * 86400)); fi
[ -n "$keyout" ] && printf -- '-----BEGIN PRIVATE KEY-----\nstub\n-----END PRIVATE KEY-----\n' > "$keyout"
[ -n "$out" ] && printf -- '-----BEGIN CERTIFICATE-----\nstub\n-----END CERTIFICATE-----\n' > "$out"
[ -n "$out" ] && printf -- '-----BEGIN CERTIFICATE-----\nstub\nSTUB_NOTBEFORE=%s\nSTUB_NOTAFTER=%s\n-----END CERTIFICATE-----\n' "$nb_epoch" "$na_epoch" > "$out"
exit 0
STUB
@@ -210,6 +254,8 @@ run_case() {
STUB_COUNTER_DIR="$WORK/counters-$name" \
STUB_SYSTEMCTL_LOG="$CASE_SYSTEMCTL_LOG" \
STUB_SYSTEMCTL_FAILED_UNITS="${STUB_SYSTEMCTL_FAILED_UNITS:-}" \
STUB_OPENSSL_NOT_BEFORE="${STUB_OPENSSL_NOT_BEFORE:-yes}" \
FIRST_BOOT_SECRETS_NOW="${FIRST_BOOT_SECRETS_NOW:-}" \
bash "$SCRIPT" > "$WORK/$name.out" 2> "$WORK/$name.err"
CASE_RC=$?
set -e
@@ -414,6 +460,72 @@ else
bad "Dockerfile heredoc quoting ->$c7"
fi
# ── Case 8: a cert minted under a wrong clock is detected and repaired ───
# The failure fail-closed cannot catch, because generation SUCCEEDS. This unit
# runs before chrony has corrected the clock; on a node with a dead RTC,
# `openssl req -x509` stamps a notBefore years out. Clock ahead -> clients
# reject the cert as "not yet valid"; clock behind -> notAfter is already in
# the past once time syncs. The old code would have marked the node done and
# never revisited it.
#
# Run 1 mints under a clock set to 2013 (a classic dead-RTC value). The node
# must still be usable — keys installed, marker set, exit 0 — but the bad dates
# must be recorded, not blessed.
# Run 2 is the same node after chrony fixes the clock: the cert must be
# regenerated with sane dates and the record cleared, with nobody at a console.
# Run 3 proves the anti-spin guard: a third run changes nothing.
BAD_CLOCK=1370000000 # 2013-06-01, i.e. a dead RTC
GOOD_CLOCK=1785000000 # 2026-07-25, inside the plausible window
c8=""
FIRST_BOOT_SECRETS_NOW="$BAD_CLOCK" run_case clock ok ok stripped
[ "$CASE_RC" -eq 0 ] || c8="$c8 run1-exit-nonzero"
[ -s "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" ] || c8="$c8 run1-no-cert-node-unusable"
[ -s "$CASE_ROOT/etc/ssh/ssh_host_ed25519_key" ] || c8="$c8 run1-no-ssh-key"
[ -f "$CASE_ROOT/var/lib/archipelago/.secrets-regenerated" ] || c8="$c8 run1-marker-missing"
grep -q 'failed=cert-dates' "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" 2>/dev/null \
|| c8="$c8 run1-BAD-DATES-NOT-RECORDED"
run1_nb=$(sed -n 's/^STUB_NOTBEFORE=//p' "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" 2>/dev/null)
FIRST_BOOT_SECRETS_NOW="$GOOD_CLOCK" run_case clock ok ok stripped 1
[ "$CASE_RC" -eq 0 ] || c8="$c8 run2-exit-nonzero"
run2_nb=$(sed -n 's/^STUB_NOTBEFORE=//p' "$CASE_ROOT/etc/archipelago/ssl/archipelago.crt" 2>/dev/null)
[ -n "$run2_nb" ] || c8="$c8 run2-cert-unreadable"
[ "$run2_nb" != "$run1_nb" ] || c8="$c8 CERT-NOT-REGENERATED-AFTER-CLOCK-FIX"
if [ -n "$run2_nb" ]; then
[ "$run2_nb" -ge 1767225600 ] || c8="$c8 run2-notBefore-still-below-floor"
# backdated, but not into the implausible past
[ "$run2_nb" -le "$GOOD_CLOCK" ] || c8="$c8 run2-notBefore-in-the-future"
[ "$run2_nb" -lt "$GOOD_CLOCK" ] || c8="$c8 run2-notBefore-not-backdated"
fi
[ -f "$CASE_ROOT/var/lib/archipelago/first-boot-secrets.failed" ] && c8="$c8 run2-stale-bad-date-record"
# Anti-spin. Counted, not date-compared: with a frozen clock a re-mint produces
# a byte-identical notBefore, so dates cannot tell "left alone" from
# "regenerated again". Counting mints is the only assertion that distinguishes
# them — the first version of this check compared dates and sailed straight
# past a deliberately broken anti-spin guard.
mints() { cat "$WORK/counters-$1/openssl-req.count" 2>/dev/null || echo 0; }
# A further run with a good clock and a good cert must NOT mint again.
before3=$(mints clock)
FIRST_BOOT_SECRETS_NOW="$GOOD_CLOCK" run_case clock ok ok stripped 1
[ "$(mints clock)" -eq "$before3" ] || c8="$c8 SPINNING-reminted-a-good-cert"
# And a node whose clock stays wrong must not mint a fresh bad cert on every
# timer tick — the loop the fix must not introduce.
FIRST_BOOT_SECRETS_NOW="$BAD_CLOCK" run_case clockstuck ok ok stripped
stuck1=$(mints clockstuck)
FIRST_BOOT_SECRETS_NOW="$BAD_CLOCK" run_case clockstuck ok ok stripped 1
stuck2=$(mints clockstuck)
[ "$stuck2" -eq "$stuck1" ] || c8="$c8 SPINNING-reminted-while-clock-still-wrong($stuck1->$stuck2)"
if [ -z "$c8" ]; then
ok "wrong clock: cert flagged not blessed, regenerated once time syncs, and no spin either way"
else
bad "wrong clock ->$c8"; fail_detail clock
fi
# ── Summary ───────────────────────────────────────────────────────────────
echo
echo "──────── first-boot-secrets summary ────────"