Archipelago — open-source initial import
This commit is contained in:
Archipelago
@@ -0,0 +1,244 @@
|
||||
# KEY-02 — fleet host-secret detection and rotation (F-03, deployed half)
|
||||
|
||||
Phase 10 plan 10-04. Companion to `docs/security/KEY-02-ROOTFS-EVIDENCE.md`, which covers the
|
||||
build half (10-03).
|
||||
|
||||
10-03 stopped the exposure growing: the ISO no longer bakes SSH host keys or a TLS keypair into
|
||||
the shared rootfs, and first-boot regeneration now fails closed instead of setting its completion
|
||||
marker on a failed run. That does **nothing** for nodes already in the field, which is exactly
|
||||
where the exposure sits — a node that hit the old fail-open path is running the SSH host key and
|
||||
TLS private key that every downloader of that ISO also holds, and it will never try again.
|
||||
|
||||
This document records the two human decisions that govern the deployed half.
|
||||
|
||||
---
|
||||
|
||||
## D-06 rotation trigger
|
||||
|
||||
**Chosen option: `detect-report-then-apply`** — recorded 2026-08-02.
|
||||
|
||||
Verbatim option id as written in `10-04-PLAN.md`: **`detect-report-then-apply`**
|
||||
("Detect and report on boot; rotate only when an operator runs the script with an explicit apply
|
||||
flag").
|
||||
|
||||
### Why
|
||||
|
||||
Rotating an SSH host key is one-way. Every `known_hosts` entry for that node breaks, on every
|
||||
machine that has ever connected to it, and the old private key is destroyed by the swap. The
|
||||
fleet is reached over Tailscale for day-to-day work and several nodes are remote — `.228` is at
|
||||
a remote site and is in real use (CLAUDE.md). `auto-on-boot` would fire that rotation on many
|
||||
nodes simultaneously during an OTA rollout, with no advance notice and no operator holding the
|
||||
new fingerprints. A node whose only access path is SSH and whose tooling pins the host key
|
||||
becomes unreachable until someone clears the entry; a rotation that fails partway on a remote
|
||||
node needs physical console access to recover, which for `.228` means a site visit.
|
||||
|
||||
Against that, the cost of `detect-report-then-apply` is that exposure persists on any node whose
|
||||
operator does not act. That cost is bounded by making the verdict **visible**: detection runs at
|
||||
boot on every node and the verdict reaches `system.stats`, so an exposed node shows up in the
|
||||
dashboard without shell access. The exposure becomes measured rather than assumed, and the list
|
||||
of nodes still to rotate is a fact on a screen rather than a guess.
|
||||
|
||||
This also matches the project's standing policy that changes are verified on the dev pair
|
||||
(archi-dev-box + x250-dev) before they reach the fleet (CLAUDE.md, `feedback_dev_pair_before_ota`).
|
||||
A rotation that fires unattended on first boot after an OTA cannot be dev-paired — by the time it
|
||||
has been observed on the dev pair it has already run everywhere.
|
||||
|
||||
### What this decision binds
|
||||
|
||||
- `scripts/security/host-secrets-audit.sh` defaults to `--detect`, which is read-only.
|
||||
- `--apply` **without** `--yes` prints its plan and exits 0 having touched nothing, so a mistyped
|
||||
invocation is inert.
|
||||
- `image-recipe/configs/archipelago-host-secrets-audit.service` ships in **detect-only** mode.
|
||||
It contains no apply path. Making the boot unit rotate would require editing the unit, which is
|
||||
a deliberate act, not a default.
|
||||
- `--apply --yes` refuses to do anything unless the detect pass returned `shared`. A node whose
|
||||
verdict is `per-node` cannot have its keys rotated by this script even by explicit command —
|
||||
the guard against "operator runs it on the wrong node" is structural, not procedural.
|
||||
|
||||
### Consequence recorded honestly
|
||||
|
||||
Any node whose verdict comes back `shared` and which is never revisited stays exposed
|
||||
indefinitely. The mitigation is the visibility, not the automation. The list under
|
||||
"Nodes with a `shared` verdict, deliberately not rotated" below exists so that no such node is
|
||||
quietly forgotten, and it is part of this plan's acceptance criteria that the list is kept.
|
||||
|
||||
---
|
||||
|
||||
## How a node decides
|
||||
|
||||
Four on-disk signals, evaluated in this precedence order by
|
||||
`scripts/security/host-secrets-audit.sh --detect`. Every verdict carries the evidence strings
|
||||
that produced it, and each evidence string names the file it was read from.
|
||||
|
||||
| # | Signal | Source |
|
||||
|---|---|---|
|
||||
| 1 | mtime of each host key / the TLS key against the first-boot anchor | `/var/lib/archipelago/.secrets-regenerated`, falling back to `/root/.luks-archipelago.key` then `/etc/machine-id` |
|
||||
| 2 | The fail-open fingerprint: marker present **and** a `WARNING:` line in the first-boot log | `/var/log/archipelago-first-boot-secrets.log` |
|
||||
| 3 | 10-03's durable failure record | `/var/lib/archipelago/first-boot-secrets.failed` |
|
||||
| 4 | Rootfs provenance | `/opt/archipelago/rootfs-identity-stripped` |
|
||||
|
||||
Verdicts: `per-node`, `shared`, `fail-closed-missing`, `unknown`.
|
||||
|
||||
**`per-node` is never reported on the strength of an absent signal.** With no anchor at all the
|
||||
verdict is `unknown`, and while a durable failure record stands the verdict is `unknown` rather
|
||||
than `per-node` — the node's own generator most recently reported failure, so a clean-looking
|
||||
mtime is not evidence of success.
|
||||
|
||||
Signal 4 changes the meaning of missing material rather than adding to the shared/per-node
|
||||
question: on a node flashed from a 10-03-or-later ISO the rootfs shipped identity-free, so an
|
||||
absent host key is a **fail-closed** state (generation never succeeded), not a shared one.
|
||||
|
||||
---
|
||||
|
||||
## C-3 — per-node host key and TLS uniqueness
|
||||
|
||||
Audit checklist item C-3 (`docs/security/ENTROPY-SEED-AUDIT-2026-07-31.md` §855), described
|
||||
there as "the highest-value check here".
|
||||
|
||||
### Status: **FAILED — with finding.** Recorded 2026-08-02.
|
||||
|
||||
> **This section names live fleet nodes that are still running shared key material.
|
||||
> Review it before this repository is made public** (`docs/OPEN-SOURCE-READINESS-PLAN.md`).
|
||||
> Digests below are truncated; the fingerprints of public keys are public data — every SSH
|
||||
> handshake offers them — but there is no reason to make a target list convenient.
|
||||
|
||||
**Three distinct live fleet nodes share all three of their SSH host keys. Two of those three
|
||||
also share their TLS certificate, and therefore their TLS private key.** This is not a
|
||||
theoretical exposure: it is F-03 in production, today.
|
||||
|
||||
#### Method
|
||||
|
||||
Gathered **remotely and read-only** — no node was logged into, nothing was written to any node,
|
||||
nothing was rotated. Host keys came from `ssh-keyscan`, which is what every SSH client does
|
||||
before it decides whether to trust a host, and certificates from an anonymous TLS handshake:
|
||||
|
||||
```bash
|
||||
ssh-keyscan -T 6 <node> | ssh-keygen -lf -
|
||||
openssl s_client -connect <node>:443 </dev/null 2>/dev/null \
|
||||
| openssl x509 -noout -fingerprint -sha256 -subject
|
||||
```
|
||||
|
||||
This is a deliberately weaker instrument than the checklist's on-node commands, and it was chosen
|
||||
because it needs no access and can therefore cover the whole reachable fleet rather than two
|
||||
nodes. What it can prove is exactly the FAIL condition: *any fingerprint appearing on two nodes*.
|
||||
|
||||
#### Result
|
||||
|
||||
| Node label | SSH host keys (ECDSA/ED25519/RSA, truncated) | TLS cert sha256 (truncated) | Cert CN |
|
||||
|---|---|---|---|
|
||||
| `archipelago-1` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
|
||||
| `archy-x250-beta` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `62:F6:A6:02…` | `archipelago` |
|
||||
| `archipelago` | `8WJplzKW…` / `lQgRXZ1n…` / `ym+gMOio…` | `7C:6B:CD:98…` | `austin-sapien` |
|
||||
| `archipelago-5` | `/bmgd6jS…` / `SpaNfLLf…` / `hhVFABi3…` | `95:FE:EB:C7…` | `archipelago.local` |
|
||||
| `archi-dev-box` | `8hFU7QGM…` / `GAxNAcgX…` / `Tv7AfaVp…` | (no :443 listener) | — |
|
||||
| `archy-dev-pa` | `JtD/RM0a…` / `XD2A5OVL…` / `esIBpbWk…` | not probed | — |
|
||||
| `framework-pt` | `oicpsj3Y…` / `zxA1/kRU…` / `oxi+tMli…` | `88:85:CE:CC…` | `framework-pt` |
|
||||
| `shorty-s` (`.228`) | `YVsgrv8M…` / `D/5n851i…` / `YMFLUerk…` | `4D:98:D4:9B…` | `shorty-s` |
|
||||
|
||||
Unreachable at scan time, so **UNVERIFIED**: `archy-x250-dev`, `archy-x250-pa`, `archy-x250-r2`,
|
||||
`quantumterminal`.
|
||||
|
||||
#### That the three are genuinely different machines, not one host seen three times
|
||||
|
||||
The obvious alternative explanation for identical host keys is a single machine registered on the
|
||||
tailnet more than once. Ruled out:
|
||||
|
||||
- All three answered a live TCP connection on port 22 within the same minute. One `tailscaled`
|
||||
instance serves one tailnet identity, so three simultaneously-live addresses are three hosts.
|
||||
- `tailscale ping` resolves them to **different physical endpoints**: `archy-x250-beta` answers
|
||||
from `178.38.147.13` (and over the Frankfurt DERP), while `archipelago-1` and `archipelago`
|
||||
answer from `45.20.199.86` on different source ports — a different continent for the first,
|
||||
and two distinct machines behind one NAT for the other two.
|
||||
- They are owned by different tailnet accounts.
|
||||
|
||||
#### Why `archipelago` has a different TLS cert but the same SSH keys
|
||||
|
||||
Its cert CN is `austin-sapien`, not the image default `archipelago`. That is the signature of a
|
||||
node that was **renamed** through `server.set-name`, which re-mints the TLS cert via
|
||||
`regenerate_tls_cert()` so the SAN matches the new hostname — and touches nothing else.
|
||||
|
||||
This is worth stating plainly because it is a trap: **TLS uniqueness alone is not evidence that
|
||||
a node's key material is per-node.** Any renamed node gets a unique certificate for free while
|
||||
its SSH host keys stay exactly as the image shipped them. Had C-3 been checked on TLS
|
||||
fingerprints only, `archipelago` would have looked clean. The SSH host key is the reliable
|
||||
signal, and this is why the audit script treats the two classes separately and reports which one
|
||||
is shared rather than issuing a single node-level verdict.
|
||||
|
||||
#### What this does NOT establish — UNVERIFIED
|
||||
|
||||
| Claim | Status | Evidence still needed |
|
||||
|---|---|---|
|
||||
| The three nodes were flashed from the **same ISO** | UNVERIFIED | Not required for the FAIL — shared host keys are the exposure however they got there — but the ISO build id would tell us how many other downloads carry the same keys. Needs on-node `/opt/archipelago/` provenance. |
|
||||
| The audit script's verdict on those three nodes | UNVERIFIED | `sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect` on each. Requires the OTA carrying this plan's runtime payload to land, or the script to be hand-staged. Predicted `shared`; predicted is not observed. |
|
||||
| A rotation preserves the operator's own session | UNVERIFIED **on hardware** | Checkpoint steps 4–6: run `--apply --yes` on one disposable node from a session you are willing to lose, confirm that session survives, confirm a second connection shows the expected mismatch. The harness proves the script's ordering and its abort path; it cannot prove that `systemctl reload ssh` keeps a real forked session alive. |
|
||||
| `host_secrets` reaches `system.stats` on a real node | UNVERIFIED | Needs a build carrying this plan deployed to the dev pair, then a `system.stats` call. Proven in unit tests against the file contract only. |
|
||||
| The four unreachable nodes | UNVERIFIED | Re-run the scan when they come back online. |
|
||||
|
||||
#### Consequence
|
||||
|
||||
`archipelago-1`, `archy-x250-beta` and `archipelago` are a **confirmed live F-03 instance**.
|
||||
Anyone holding a copy of the ISO these nodes were flashed from holds their SSH host private keys,
|
||||
and for the first two, their TLS private key as well — enough for undetectable SSH host
|
||||
impersonation and transparent MITM of the web UI.
|
||||
|
||||
None of them was rotated as part of this verification, and that is deliberate: this checkpoint
|
||||
verifies, it does not remediate, and remediating a node inside a verification task is how a
|
||||
verification task takes a node offline. They are recorded below.
|
||||
|
||||
---
|
||||
|
||||
## Nodes with a `shared` verdict, deliberately not rotated
|
||||
|
||||
Any node that reports `shared` and is not rotated in the same session MUST be added here with the
|
||||
date and the reason, so that the standing consequence of `detect-report-then-apply` is a visible
|
||||
list rather than an assumption.
|
||||
|
||||
| Node label | Date detected | Why not rotated | Next step |
|
||||
|---|---|---|---|
|
||||
| `archipelago-1` | 2026-08-02 | Detected by remote fingerprint comparison during C-3, not by an operator running the script. In real use; rotating it inside a verification task is exactly what the task forbids. | Stage the script, run `--detect`, then rotate from a session the operator is willing to lose. |
|
||||
| `archy-x250-beta` | 2026-08-02 | Same. Also shares its **TLS private key** with `archipelago-1`, so it is the more urgent of the two. Reached over a DERP relay from another continent — the least recoverable node in the set if a rotation goes wrong. | Rotate from physical or console access if available; otherwise rotate TLS first, confirm, then SSH. |
|
||||
| `archipelago` | 2026-08-02 | Same. TLS is already unique (the node was renamed, which re-mints the cert); only its SSH host keys are shared. | `--apply --yes` will rotate SSH only — the detect pass flags the classes separately, so this node's already-unique TLS pair is left alone. |
|
||||
|
||||
**Nobody has been told their `known_hosts` is about to break.** Three nodes here are in real use;
|
||||
the rotation is one-way and every existing entry for them dies with it. Sequencing that is an
|
||||
operator decision, which is the whole content of D-06.
|
||||
|
||||
---
|
||||
|
||||
## Operator runbook — rotating one node
|
||||
|
||||
Run this from a session you are willing to lose, on **one node at a time**. Never on `.228` or
|
||||
any node in real use without arranging access recovery first.
|
||||
|
||||
```bash
|
||||
# 1. Detect. Read-only; safe on any node, including production.
|
||||
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --detect
|
||||
cat /var/lib/archipelago/host-secrets-audit.json
|
||||
|
||||
# 2. Dry run. Prints the plan, touches nothing, exits 0.
|
||||
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply
|
||||
|
||||
# 3. Rotate. Only proceeds if the verdict is `shared`.
|
||||
sudo /opt/archipelago/scripts/security/host-secrets-audit.sh --apply --yes
|
||||
|
||||
# 4. WITHOUT closing that session, prove it survived:
|
||||
echo still-here
|
||||
|
||||
# 5. From a second terminal, expect a host-key mismatch warning. That is the
|
||||
# correct outcome. Update known_hosts against the fingerprints printed by
|
||||
# step 3 (also in /var/lib/archipelago/host-key-rotation.json), never by
|
||||
# blindly accepting whatever is offered.
|
||||
ssh-keygen -R <node>
|
||||
ssh <node>
|
||||
|
||||
# 6. The web UI will present a new self-signed cert. A fresh browser trust
|
||||
# prompt is expected and is the correct outcome.
|
||||
```
|
||||
|
||||
The script reloads sshd rather than restarting it. A reload re-execs the listener while
|
||||
already-forked session children keep running, which is why the operator's own SSH session
|
||||
survives its own rotation. `restart` would kill it, and on a remote node with no console that is
|
||||
unrecoverable.
|
||||
|
||||
Old fingerprints are written to `/var/lib/archipelago/host-key-rotation.json` **before** the
|
||||
swap, so an operator who loses access anyway can still identify what changed.
|
||||
Reference in New Issue
Block a user