archy/.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md

---
phase: 01-federation-mesh-hardening
plan: 16
subsystem: security
tags: [secrets, bcrypt, fedimint, migration, rotation, reconcile]

requires:
  - phase: 01-federation-mesh-hardening
    provides: "01-11's KNOWN_DEFAULT_GATEWAY_HASHES denylist, ensure_gateway_credential, gateway_bcrypt_hash and the atomic 0600 write_secret — rotation reuses all of it and adds no new generation or file-writing code"
provides:
  - "rotate_compromised_gateway_credential(secrets_dir) -> Result<bool>: denylist-exact detection plus rotation of a shipped gateway credential"
  - "Self-healing on the existing reconcile tick, so an affected node rotates without operator action and without a hand-rolled container teardown"
affects: [fedimint-gateway, container-secrets, reconcile]

tech-stack:
  added: []
  patterns:
    - "Rotate by changing the secret, not by touching the container: writing the new credential changes the resolved secret env, which changes secret_env_hash, which the drift check reads as a container-label mismatch — so the platform's own recreate path rebuilds the container around unchanged data, ports, volumes and name."
    - "Denylist-exact detection: rotate only on an exact match against known-compromised values, never on 'unrecognised'. An operator's deliberately-set credential is unrecognised too."

key-files:
  created: []
  modified:
    - core/archipelago/src/container/secrets.rs
    - core/archipelago/src/container/prod_orchestrator.rs

key-decisions:
  - "Bcrypt generation was factored out of ensure_one's Bcrypt arm into write_bcrypt_pair(dir, name), which both ensure_one and rotation call. 01-11 had left that arm inline, and rotation cannot reuse ensure_gateway_credential directly because ensure_one's idempotent fast path returns early when the file is present and non-empty — which is exactly the case rotation must act on."
  - "The rotation call is gated on `manifest.app.id == \"fedimint-gateway\"` rather than running for every app on every tick. It hangs off resolve_dynamic_env, immediately after ensure_generated_secrets, as the plan specified."
  - "Errors propagate (`?`) rather than being logged-and-continued: write_secret's atomic temp-file-plus-rename leaves the previous credential intact on failure, so surfacing the error is strictly safer than proceeding with a half-rotated gateway."
  - "No boot-specific wiring was added — see the boot-reconciler finding below."

requirements-completed: []

coverage:
  - id: D1
    description: "A node carrying the shipped default rotates itself onto a unique credential without operator action"
    requirement: "FED-07"
    verification:
      - kind: unit
        ref: "core/archipelago/src/container/secrets.rs#rotates_a_denylisted_gateway_credential"
        status: pass
      - kind: manual_procedural
        ref: "Task 2 checkpoint, archi-dev-box 2026-08-01 — the credential FILE rotates correctly (~15s after restart, fresh unique value, 0600), but the RUNNING gateway keeps the pre-rotation credential: Quadlet rewrites the unit without restarting it, and the gateway is classified restart-sensitive so drift is detected and deliberately ignored on every tick"
        status: fail
    human_judgment: true
  - id: D2
    description: "A node already carrying a unique credential is left completely alone; detection never fires on merely-unrecognised values"
    requirement: "FED-07"
    verification:
      - kind: unit
        ref: "…#leaves_a_unique_gateway_credential_alone, …#leaves_an_unrecognised_credential_alone"
        status: pass
    human_judgment: false
  - id: D3
    description: "Rotation runs at most once per affected node; later ticks detect nothing and change nothing"
    requirement: "FED-07"
    verification:
      - kind: unit
        ref: "…#rotation_is_idempotent"
        status: pass
    human_judgment: false
  - id: D4
    description: "Rotation replaces one credential pair and nothing else — no other secret, and no app data, is touched"
    requirement: "FED-07"
    verification:
      - kind: unit
        ref: "…#rotation_touches_no_other_secret (four bystander secrets asserted byte-identical)"
        status: pass
      - kind: other
        ref: "git diff of prod_orchestrator.rs contains zero added rm -f / remove_dir_all / podman rm / chown"
        status: pass
    human_judgment: false
  - id: D5
    description: "The rotation is announced in the node's logs without ever printing the credential"
    requirement: "FED-07"
    verification:
      - kind: other
        ref: "The info! line interpolates self.secrets_dir and the secret NAME only; no value is in scope at the call site (rotate returns bool, not the credential)"
        status: pass
      - kind: manual_procedural
        ref: "Task 2 step 3, archi-dev-box 2026-08-01 — one info line fired, naming /var/lib/archipelago/secrets/fedimint-gateway-hash.pw; no credential value anywhere in the log"
        status: pass
    human_judgment: true
  - id: D6
    description: "Generation where no credential exists stays ensure_gateway_credential's job"
    requirement: "FED-07"
    verification:
      - kind: unit
        ref: "…#no_op_when_no_gateway_credential_exists"
        status: pass
    human_judgment: false

duration: 140min
completed: 2026-08-01
status: task-1-complete-checkpoint-FAILED-recreate-does-not-fire
---

# Phase 1 Plan 16: Rotate Existing Installs Off the Shipped Gateway Credential (FED-07) Summary

**Task 1 rotates the credential correctly and was proven to do so on a real node. But the checkpoint DISPROVED the assumption it rests on: the rotated credential never reaches the running container, because the Quadlet path does not restart units and the gateway is classified restart-sensitive. FED-07 remains open.**

## Status

**FED-07 is NOT closed, and this plan alone cannot close it.** The checkpoint ran on archi-dev-box on
2026-08-01 and found that rotation does not propagate to the running gateway. A follow-up plan is
required — see the checkpoint result below.

## Accomplishments

- `rotate_compromised_gateway_credential(secrets_dir) -> Result<bool>` in `container::secrets`:
  reads the canonical hash file, returns `Ok(false)` for absent/unreadable/unique/unrecognised, and
  only on an **exact** denylist match writes a fresh pair and returns `Ok(true)`.
- `write_bcrypt_pair(dir, name)` factored out of `ensure_one`'s `Bcrypt` arm so there is exactly one
  bcrypt-generation implementation, called by both generation and rotation.
- Wired into `resolve_dynamic_env` beside `ensure_generated_secrets`, gated on the gateway's app id,
  with an info-level announcement that names the *path* to the new plaintext and never the value.
- Six new tests covering rotate-on-denylisted (including 0600 modes and that the `.pw` sibling
  verifies against the new hash), no-op-on-unique, no-op-on-unrecognised, no-op-on-absent,
  idempotence, and four bystander secrets left byte-identical.

## Findings the plan asked for

### Boot reconciler needs no separate call

`boot_reconciler` calls `reconcile_all()` → `reconcile_all_with_mode()` → per-manifest
`ensure_running_with_mode()` (prod_orchestrator.rs:1714) → `resolve_dynamic_env()`
(prod_orchestrator.rs:1914) → the rotation call. `install_fresh` reaches it by the same route.
So boot and reconcile funnel through one chokepoint and **no boot-specific wiring was added**;
`boot_reconciler.rs` is not in `files_modified`.

### ✅ RESOLVED 2026-08-02: the carve-out closes it, proven on the same node

The failure documented below was fixed and re-verified on archi-dev-box.

**The fix.** Rotation now records the app id in a `credential_rotated` set on the orchestrator, and
the drift check consumes that flag to recreate the container **even when the app is
restart-sensitive**, with a `WARN` naming the reason. It deliberately mirrors the published-port
carve-out sitting a few lines above it in the same function, which already makes exactly this trade
for exactly this reason: a container that is already non-functional (there) or already compromised
(here) is not protected by leaving it running. No teardown was hand-rolled — the existing recreate
path does the work.

**The design call.** Restart-sensitivity protects *working* services. A gateway answering to a
credential published in this repository is not working, it is compromised, and an attacker with
gateway admin can drain Lightning liquidity. Indefinite exposure loses to a few seconds of restart.
The alternative — rotate but only raise an operator alert — was rejected because the monitoring
system fires alerts from metric thresholds only (`check_alerts(&MetricSnapshot)`), so it would have
needed new event-alert plumbing to deliver something strictly weaker than just fixing it.

**Before / after on the same box, same scenario:**

| | Before (15:51, first checkpoint) | After (06:39, re-run) |
|---|---|---|
| Credential file | rotated ✅ | rotated ✅ |
| Container recreated | ❌ never — same PID 25 min later | ✅ 4s after rotation, PID 3923125 → 148426 |
| Running credential | the compromised default | matches the file (`636d6031…`) |
| Log | "leaving running restart-sensitive app untouched" ×4 and counting | "recreating restart-sensitive app: its admin credential was rotated off a publicly known default…" |

Post-recreate state: container healthy, **same name**, **same ports** (8176/9737), `gatewayd.db`
intact at 18 files with `IDENTITY` present, credential `0600 archipelago:archipelago`, 32 containers
untouched, and **zero repeat rotations** on subsequent ticks (the rotated value is not denylisted, so
it is self-terminating — T-01-73 holds).

Covered by three new tests: `rotating_a_compromised_credential_flags_the_app_for_recreate`,
`a_unique_credential_does_not_flag_the_app`, `a_second_pass_does_not_re_flag_the_app`.

---

### ⛔ ORIGINAL CHECKPOINT RESULT (2026-08-01) — retained: this is what the fix had to defeat

**Run on archi-dev-box, 2026-08-01.** The code-reading conclusion below was **wrong in practice**,
which is exactly why the plan made this a blocking checkpoint. The rotation works; the propagation
to the running container does not.

Observed: rotation fired ~15s after restart, wrote a fresh unique credential, and logged correctly.
But **25 minutes later the running gatewayd process was still using the pre-rotation credential.**
`/proc/<pid>/environ` for PID 1 held the old value while the file and the podman secret held the new
one. The orchestrator says why, in its own log lines:

```
Quadlet unit drift-synced — file rewritten, .service NOT restarted
    (operator restart picks up new config)                     app_id=fedimint-gateway
container drift detected during boot reconcile;
    leaving running restart-sensitive app untouched            app_id=fedimint-gateway
```

Two independent guards, both deliberate:

1. **The Quadlet path rewrites the `.container` unit but never restarts the `.service`.** The unit
   file was rewritten at 15:51:36 (same second as the rotation) carrying the new
   `secret-env-hash=de9870c642a515f7` label — so the definition updated correctly. Systemd does not
   apply a changed unit to a running container without a restart.
2. **`fedimint-gateway` is classified restart-sensitive**, so the drift check *detects* the change on
   every reconcile tick and then deliberately leaves the container alone. That line repeated at
   15:51, 15:53, 15:54, 15:56 — it will repeat forever.

**Consequence: on a real affected node, rotation makes the credential file unique but the gateway
keeps answering to the compromised one indefinitely** — until an unrelated reboot or a manual
restart. Worse, the operator reading `fedimint-gateway-hash.pw` gets a password the running gateway
does not accept, which is the lockout risk T-01-77 inverted.

Confirmed the fix works when applied: `systemctl --user restart fedimint-gateway.service` produced a
new PID whose `FEDI_HASH` is the rotated value, container healthy, same name, same ports, marker file
and `gatewayd.db` intact.

**Deliberately NOT hand-rolled.** The plan's action says: "If that path does not fire for this app for
some reason you discover, do not hand-roll a remove-and-run; stop and record what you found." So this
is recorded, not patched. The fix belongs in a follow-up and has to answer a real design question:
a compromised credential is arguably the one case that should override restart-sensitivity — or,
failing that, the rotation must raise an operator-facing "restart required" alert rather than logging
into the void.

### What the checkpoint DID prove

| Step | Result |
|---|---|
| 1. State recorded | Node was CLEAN; affected state seeded deliberately (plan's Planner Assumption) |
| 2. Deploy | `install -m0755` to `/usr/local/bin/archipelago` + `systemctl restart archipelago`; rollback kept at `archipelago.bak-pre-fed07` |
| 3. Rotation announced | ✅ One info line, fired once, names the `.pw` path, **no credential value in the log** |
| 4. Credential unique | ✅ Third distinct value (not the default, not the pre-test original), `0600 archipelago:archipelago` |
| 5. Data preserved | ✅ Marker file and all 13 `gatewayd.db` files incl. `IDENTITY` intact; same container name; same ports 8176/9737 |
| 5. Container recreated | ❌ **FAILED — see above** |
| 6. Auth proof | ⚠️ Not obtainable: `gateway-cli` in this image returns the same "Invalid request" for a correct and an incorrect password, so it cannot distinguish them. Substituted PID-1 `environ` comparison, which is stronger evidence of *which* credential is in force. |
| 7. Fresh install differs | ⏸ Not run |
| 8. `run-gate.sh` | ⏸ Not run |

Also proved incidentally: **restarting `archipelago` does not kill containers on this box** — 29/29
and later 31/31 survived, and the orchestrator logged "Adopted 31 existing container(s)". The
CLAUDE.md "restart SIGKILLs containers" rule does not apply under `ARCHIPELAGO_USE_QUADLET_BACKENDS=true`
with podman in the user slice (the service is `system.slice`/`KillMode=control-group`; the containers
live in `user-1000.slice/…/libpod-*`, a different cgroup entirely).

### Original code-reading conclusion (retained — it is what the checkpoint disproved)

`resolve_dynamic_env` computes `secret_env_content_hash(&secret_bearing)` over the resolved
secret-bearing env and stores it as `manifest.app.container.secret_env_hash`
(prod_orchestrator.rs:3309). The drift check (prod_orchestrator.rs:3374) inspects the running
container's `SECRET_ENV_HASH_LABEL` and returns "drifted" when it differs from the expected hash,
which drives the existing recreate. The gateway's `FEDI_HASH` comes from the rotated file, so a
rotation necessarily changes that hash and therefore the label comparison.

**This is a code-reading conclusion. It has not been observed firing on a node** — that is Task 2
step 5, and it is the single most important thing the checkpoint proves.

### Operator recovery: the surface exists but does NOT cover this app — a real gap

- The UI path is live: `Apps.vue` calls `package.credentials` with an `app_id` before launching an
  app and renders a credentials modal from the response.
- The backend, `handle_package_credentials` in
  `core/archipelago/src/api/rpc/package/install.rs:2093`, is a hardcoded per-app if-chain covering
  **only `filebrowser` and `photoprism`**. Every other app, including `fedimint-gateway`, falls
  through to `Ok(json!({ "credentials": [] }))`.
- **Consequence:** after rotation the operator has no in-UI way to obtain the new gateway password.
  The recovery path is the file the log line names: `/var/lib/archipelago/secrets/fedimint-gateway-hash.pw`
  (0600, service user), readable over SSH.
- **Gap owner:** `handle_package_credentials` in `core/archipelago/src/api/rpc/package/install.rs`.
  Adding a `fedimint-gateway` arm that reads the `.pw` sibling would close it; the UI needs no change.
  Deliberately not done here — this plan's `files_modified` is scoped to two files, and that handler
  belongs to the app-credentials surface, not to FED-07's rotation.

## Adjacent finding — NOT part of this plan, raised deliberately

`apps/photoprism/manifest.yml:35` sets `PHOTOPRISM_ADMIN_PASSWORD=archipelago`, and
`handle_package_credentials` hands that same literal back to the UI. That is a shipped default
credential in a manifest — the same class of defect as FED-07, on a different app. Every node running
PhotoPrism answers to `admin` / `archipelago`.

It is out of scope here (this plan is the gateway migration) and was not touched. It wants its own
requirement and plan, and probably the same treatment: a `generated_secrets` entry plus a denylist
entry for the shipped value.

## Deviations from Plan

### Bcrypt generation had to be factored out first

**Found during:** Task 1
**Issue:** The plan says rotation should "generate a replacement pair through the same helper
`ensure_gateway_credential` uses". 01-11 never actually created such a helper — it left the bcrypt
arm inline in `ensure_one` and had `ensure_gateway_credential` call `ensure_one`. Rotation cannot
call `ensure_gateway_credential`, because `ensure_one`'s idempotent fast path returns early when the
target files are present and non-empty, which is precisely the state rotation acts on.
**Resolution:** Extracted `write_bcrypt_pair(dir, name)` from the `Bcrypt` arm; `ensure_one` and
rotation both call it. Still exactly one generation implementation, which is what the instruction was
protecting.
**Files modified:** `core/archipelago/src/container/secrets.rs`

## Known Stubs

None.

## Threat Flags

- **T-01-72 (critical, EoP)** — mitigated in code, **not yet proven on a node**. Task 2 step 6 (old
  credential rejected, new one accepted) is the proof and has not been run.
- **T-01-73 (DoS, rotation loop)** — mitigated and unit-tested: the rotated value is not on the
  denylist, so the next tick is a no-op (`rotation_is_idempotent`).
- **T-01-74 (info disclosure)** — mitigated structurally: `rotate_compromised_gateway_credential`
  returns `bool`, so the credential is not even in scope at the logging call site.
- **T-01-75 (tampering / data loss)** — mitigated: no teardown primitives added (grep-verified), the
  recreate goes through `secret_env_hash`. On-node data-survival check is Task 2 step 5, not run.
- **T-01-76 (repudiation — signing off without exercising rotation)** — **live risk, unresolved.**
  Whether archi-dev-box is affected or already clean is still unknown; the plan requires declaring
  which case it is and deliberately seeding the old value if the node is clean.
- **T-01-77 (operator lockout)** — partially mitigated: the plaintext exists at a named 0600 path and
  the log line points at it, but there is no UI retrieval path (see the gap above).
- **T-01-SC** — no crates added.

## Self-Check

- CONFIRMED: `cargo test -p archipelago secrets` → **16 passed, 0 failed** (the `container::secrets`
  module holds 14 `#[test]` fns, all six new rotation cases among them:
  `rotates_a_denylisted_gateway_credential`, `leaves_a_unique_gateway_credential_alone`,
  `leaves_an_unrecognised_credential_alone`, `no_op_when_no_gateway_credential_exists`,
  `rotation_is_idempotent`, `rotation_touches_no_other_secret`)
- FOUND: `rotate_compromised_gateway_credential` in `secrets.rs` (definition + 5 test uses)
- FOUND: exactly 1 non-comment reference in `prod_orchestrator.rs`
- CONFIRMED: 0 added teardown primitives (`rm -f` / `remove_dir_all` / `podman rm` / `chown`) in the
  `prod_orchestrator.rs` diff
- CONFIRMED: `cargo fmt --check -p archipelago` clean. It was **not** clean before this plan —
  `install.rs` carried drift introduced by 01-11's commit (`42652547`), fixed here. That check has
  blocked the release gate before (`37d293be`), so it is worth keeping green rather than discovering
  at ship time.
- CONFIRMED: `cargo test -p archipelago` (after `cargo clean -p archipelago`) — **1008 passed, 1
  failed**. The failure is `container::boot_reconciler::tests::second_pass_fires_after_interval`, the
  same wall-clock-timed test (50ms tick) that was flaky during 01-11; re-run in isolation it is
  **4 passed / 0 failed in 0.46s**. `boot_reconciler.rs` is untouched by this plan.
- **NOT RUN:** Task 2's eight-step on-node checkpoint, and `tests/lifecycle/run-gate.sh`

### A false alarm worth recording, because it cost an hour

An intermediate full-suite run reported `credentials::operations::tests::test_list_credentials_filter_by_did`
failing with "invalid utf-8 sequence of 1 bytes from index 2" — an identity-credentials test in a
module this plan does not touch, which had passed in the 01-11 run two hours earlier.

Cause: **corrupted build artifacts, not a regression.** Two duplicate `cargo test` runs had been
started against the same workspace lock and one was `SIGTERM`ed to free it. The next compile surfaced
`rust-lld: error: undefined hidden symbol` — precisely the incremental-cache corruption CLAUDE.md
documents. After `cargo clean -p archipelago` the credentials test passes and the only failure is the
known timing flake above.

Lesson for the next executor on this box: do not kill an in-flight `cargo` to free the build lock —
let it finish. A corrupted target dir produces failures in modules you never touched, which reads
exactly like a real regression and is not one.
</content>