archy/.planning/phases/01-federation-mesh-hardening/01-16-PLAN.md

---
phase: 01-federation-mesh-hardening
plan: 16
type: execute
wave: 8
depends_on: ["01-11"]
files_modified:
  - core/archipelago/src/container/secrets.rs
  - core/archipelago/src/container/prod_orchestrator.rs
autonomous: false
requirements: [FED-07]
gap_closure: true

must_haves:
  truths:
    - "A node already running a gateway on the shipped default credential rotates itself onto a unique one without an operator having to know it was affected (FED-07 migration)"
    - "Rotation preserves the gateway's data: /var/lib/archipelago/fedimint-gateway survives, and so do the container name, its ports, its volumes and its adoption identity (CLAUDE.md — migrations never destroy data)"
    - "A node already carrying a unique credential is left completely alone — detection matches the known defaults only, never 'anything I did not generate this run' (FED-07 adjacency edge)"
    - "Rotation runs at most once per affected node: after it completes, later reconcile ticks detect nothing and change nothing (FED-07 idempotence)"
    - "A rotation the operator can see: it is announced in the node's logs naming the app and that credentials changed, and it never prints the credential itself"
    - "After rotation the operator has a supported way to obtain the new gateway credential, so rotating does not lock them out of their own gateway"
    - "A rotation that cannot complete leaves the previous working state intact and reports an error rather than leaving a gateway configured against a credential nobody holds (FED-07 failure-surfacing)"
  prohibitions:
    - statement: "Rotation MUST NOT delete, move, reinitialise or chown the gateway's data directory, its Lightning backend credentials, or any other app's secrets — it replaces one credential file and lets the existing recreate path rebuild the container around unchanged data"
      category: safety
    - statement: "The rotated credential MUST NOT be written to a log line, a status RPC response, a deploy transcript, or any file outside the 0600 rootless secrets directory"
      category: privacy
    - statement: "Detection MUST NOT rotate a credential merely because it is unrecognised — only an exact match against the known-default denylist triggers rotation, so an operator who set their own credential deliberately keeps it"
      category: safety
  artifacts:
    - path: core/archipelago/src/container/secrets.rs
      provides: "Denylist-driven detection and rotation of a compromised gateway credential"
      contains: "rotate_compromised_gateway_credential"
  key_links:
    - from: core/archipelago/src/container/prod_orchestrator.rs
      to: core/archipelago/src/container/secrets.rs
      via: "the reconcile path that already materialises generated secrets also asks for compromised-credential rotation, so an existing node heals on its next tick"
      pattern: "rotate_compromised_gateway_credential"
---

<objective>
Get the nodes that are already running on the shipped gateway credential off it, without touching
their data.

Purpose: plan 01-11 stops new installs from ever taking a shipped credential, but it does nothing for
the nodes that already did. Those gateways answer to a credential published in this repository, so
until they rotate, FED-07 is only half closed — and the requirement is explicit that existing installs
carrying the default get a migration path. The repo's standing rule bounds how: migrations never
destroy data — preserve `/var/lib/archipelago/<app>`, secrets, credentials, ports and adoption
container names, and keep a rollback path.
Output: detection against the denylist plan 01-11 established, rotation through the recreate machinery
that already preserves data, an operator-visible announcement, and a sign-off on a real node.
</objective>

<execution_context>
@$HOME/.claude/gsd-core/workflows/execute-plan.md
@$HOME/.claude/gsd-core/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/STATE.md
@.planning/phases/01-federation-mesh-hardening/01-11-SUMMARY.md
@apps/fedimint-gateway/manifest.yml
</context>

## Artifacts this phase produces

Created or changed by **this plan**:

| Symbol | Kind | File |
|---|---|---|
| `rotate_compromised_gateway_credential(secrets_dir) -> Result<bool>` | new detection + rotation entry point | `core/archipelago/src/container/secrets.rs` |
| rotation call on the reconcile path | changed reconcile step | `core/archipelago/src/container/prod_orchestrator.rs` |

<tasks>

<task type="tracer" tdd="true">
  <name>Task 1: End-to-end — a node carrying the default rotates itself and keeps its data</name>
  <files>core/archipelago/src/container/secrets.rs, core/archipelago/src/container/prod_orchestrator.rs</files>
  <read_first>
    - `core/archipelago/src/container/secrets.rs` as left by plan 01-11 — the
      `KNOWN_DEFAULT_GATEWAY_HASHES` denylist, `ensure_gateway_credential`, `gateway_bcrypt_hash`, the
      shared bcrypt generation helper, and the atomic 0600 `write_secret`. Rotation reuses all of it;
      write no new generation or file-writing code.
    - `core/archipelago/src/container/prod_orchestrator.rs` around line 3227 and line 3245 — the
      comment naming the per-app generated secrets (`fmcd-password`, `fedimint-gateway-hash`, …) and
      the `crate::container::secrets::ensure_generated_secrets(&self.secrets_dir, manifest)?` call.
      This is the tick that runs on every reconcile and the natural place to hang detection.
    - `core/container/src/manifest.rs` — grep for `secret_env_hash` and read its definition and every
      use. This is the existing mechanism by which a changed secret drives a container recreate, and
      it is what makes rotation preserve data: the recreate path it feeds already keeps the data
      directory, ports, volumes and container name. Reuse it rather than stopping and removing the
      container by hand.
    - `apps/bitcoin-ui/manifest.yml` lines 8-45 — the in-repo precedent for "the password rotated, so
      the rendered bytes changed, so the container is recreated". Read it for how a rotation is
      expected to propagate on this platform.
    - `core/archipelago/src/container/boot_reconciler.rs` — enough to determine whether boot has its
      own separate path that also needs the call, or whether it funnels through the same reconcile
      step. Record the finding; if it needs the call too, add that file to `files_modified` in the
      SUMMARY.
  </read_first>
  <behavior>
    - Given a secrets dir whose gateway hash file contains a denylist entry, rotation replaces it with
      a freshly generated pair and reports that it rotated.
    - Given a secrets dir whose gateway hash is not on the denylist, rotation changes nothing and
      reports that it did not rotate — including when the value is one nobody recognises.
    - Given a secrets dir with no gateway hash at all, rotation changes nothing and reports that it did
      not rotate; generation is `ensure_gateway_credential`'s job, not rotation's.
    - Running rotation twice on the same affected dir rotates once; the second run is a no-op.
    - After rotation the new hash is not on the denylist and its `.pw` sibling verifies against it.
    - Rotation touches no file other than the gateway credential pair — every other file in the
      secrets dir is byte-identical afterwards.
  </behavior>
  <action>
    Write the tests in `secrets.rs`'s `mod tests` first and confirm they fail. Use `tempfile::tempdir`
    the way the existing tests in that module do, and seed the affected case by writing a denylist
    entry into the hash file. Include a case that seeds several unrelated secret files alongside it and
    asserts they are untouched.

    Add `pub fn rotate_compromised_gateway_credential(secrets_dir: &Path) -> Result<bool>` to
    `secrets.rs`. It reads the gateway hash file; if it is absent or unreadable it returns `Ok(false)`
    without writing; if its trimmed value is not an exact match for a denylist entry it returns
    `Ok(false)`; only on an exact match does it generate a replacement pair through the same helper
    `ensure_gateway_credential` uses and return `Ok(true)`. Because the underlying write is the
    existing atomic temp-file-plus-rename, a failure mid-rotation leaves the previous file in place —
    that is the rollback path, and it should be stated in the function's doc comment so nobody later
    "improves" it into a truncate-in-place.

    Wire it into `prod_orchestrator.rs` immediately alongside the existing `ensure_generated_secrets`
    call. When it returns `true`, log at info level that the Fedimint gateway credential was rotated
    because the node was carrying a publicly known default, that the gateway will be recreated, and
    where the operator can obtain the new one — and never log the value. Then make the recreate happen
    through the existing `secret_env_hash` change-detection path rather than by stopping or removing
    the container directly: the hash file changed, so the resolved secret env changes, so the platform's
    own recreate machinery fires with the data directory, ports, volumes and container name all
    preserved. If that path does not fire for this app for some reason you discover, do not hand-roll a
    remove-and-run; stop and record what you found, because a hand-rolled recreate is the exact
    anti-pattern CLAUDE.md names.

    Settle the operator-recovery question and record the answer. The plaintext already lands at
    `fedimint-gateway-hash.pw`, 0600, rootless. Determine whether the app-credentials surface in the UI
    (`neode-ui/src/views/Credentials.vue` and whatever RPC feeds it) already exposes per-app generated
    credentials. If it does, confirm the rotated value appears there and say so. If it does not, the
    log line must name the exact path an operator reads, and the SUMMARY must record that a UI surface
    is a gap with the file that would own it. Do not leave "how does the operator get the new password"
    unanswered — rotating a credential the user cannot retrieve is a lockout, not a fix.

    Do not change the gateway's ports, volumes, data directory, network, capabilities, health check or
    any other manifest-driven property.
  </action>
  <verify>
    <automated>cd core &amp;&amp; cargo test -p archipelago secrets 2>&amp;1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - `cd core && cargo test -p archipelago secrets` exits 0 and its output names cases for: rotates-on-denylisted, no-op-on-unique, no-op-on-absent, idempotent-second-run, and other-secrets-untouched.
    - `grep -v '^\s*//' core/archipelago/src/container/secrets.rs | grep -c 'rotate_compromised_gateway_credential'` is at least 2 (definition plus test use).
    - `grep -v '^\s*//' core/archipelago/src/container/prod_orchestrator.rs | grep -c 'rotate_compromised_gateway_credential'` equals 1.
    - `git diff -- core/archipelago/src/container/prod_orchestrator.rs | grep -ciE '^\+.*(rm -f|remove_dir_all|podman rm|chown)'` equals 0 — no hand-rolled teardown was introduced.
    - `cd core && cargo build -p archipelago` exits 0 and `cd core && cargo test -p archipelago` exits 0.
    - The SUMMARY records the boot-reconciler finding, whether the recreate fired through `secret_env_hash`, and the operator-recovery answer with its evidence.
  </acceptance_criteria>
  <done>An affected node heals itself on its next reconcile tick, once, without losing data, and the operator can still get into their gateway.</done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 2: Confirm the rotation on a real node</name>
  <what-built>
    FED-07 in full, ready to run on a node:
    - No code path in the tree can configure a Fedimint gateway with a credential that shipped with
      the repository. The five fallback sites (two in the Rust orchestrator, three in the install and
      deploy scripts) are gone, along with the plaintext password fallback in the Tailscale deploy
      path. The one surviving copy of the old hash is a denylist used only to detect it.
    - Every install now takes its credential from the per-install secret the manifest already
      declared, generated at 0600 by `container::secrets`.
    - A node that is already carrying the old default rotates itself on its next reconcile tick and
      is recreated around its existing data directory, ports and container name.
  </what-built>
  <how-to-verify>
    Run this on archi-dev-box. Note that archy-x250-dev has been offline since phase 2 — do not wait
    for it; single-node verification with the second-node gap recorded honestly is the expected
    pattern here.

    1. **Before you change anything, record the current state.** On the node, check whether the
      gateway credential file currently holds the old shipped value, and whether a gateway container
      is running. Note both. This is what tells you whether you are testing the rotation path or the
      already-clean path — say which one you got.

    2. **Deploy this phase's build to archi-dev-box only.** Use the dev-pair deploy path, not a
      release, not an OTA, and not the Tailscale alpha-tester path. Record the exact command.

    3. **Watch the rotation.** Follow the node's logs across a reconcile tick. Expected if the node was
      affected: one info line saying the gateway credential was rotated because a publicly known
      default was in use, naming where to get the new one — and no credential value anywhere in the
      log. Expected if the node was already clean: no rotation line at all.

    4. **Confirm the credential is now unique.** Read the gateway hash file on the node and confirm it
      is not the old shipped value, and that its file mode is 0600 and it is owned by the rootless
      service user, not root.

    5. **Confirm the data survived.** List `/var/lib/archipelago/fedimint-gateway` and confirm its
      contents are the same ones that were there in step 1 — the gateway's own state must not have been
      reinitialised. Confirm the container came back with the same name and the same published ports.

    6. **Confirm the gateway actually works.** Check the container is running and healthy, and that its
      admin endpoint answers. Then authenticate to it with the new credential from the path the log
      line named. Expected: the new credential works. Then try the old shipped one. Expected: rejected.

    7. **Confirm a fresh install is unique too.** If practical, uninstall and reinstall the gateway on
      the node and confirm the credential it comes up with differs from the one from step 4 — that is
      the per-install property, and it is the whole point of the requirement.

    8. **Confirm nothing else moved.** Run `tests/lifecycle/run-gate.sh` on the node (the gate runs
      on-node, never over RPC) and confirm it is still green. This plan changed orchestrator reconcile
      behaviour, which is exactly the case CLAUDE.md says to re-run the gate for. A single clean pass
      is enough here; the 5× run is Phase 3's criterion.

    If any step fails, say which numbered step and what you saw — that becomes the gap list rather than
    a re-run of the whole plan.
  </how-to-verify>
  <resume-signal>Type "approved" to sign off FED-07, or describe the issues by step number.</resume-signal>
</task>

</tasks>

## Planner Assumptions (flagged, unresolved)

- **Whether archi-dev-box is actually affected is unknown to the planner.** Its gateway may have been
  provisioned by a path that generated a unique credential. Step 1 makes the executor establish which
  case they are in and say so, rather than reporting a green run that never exercised the rotation. If
  the node is clean, the rotation path still needs proving — seed the old value into the credential
  file on the node deliberately, then re-run steps 3 to 6, and record that you did.
- **Whether the UI already exposes per-app generated credentials** was not verified by the planner.
  Task 1 makes it an explicit finding with a named owning file if it turns out to be a gap.

<threat_model>
## Trust Boundaries

| Boundary | Description |
|----------|-------------|
| gateway admin API → network | The credential being rotated is the only gate on Lightning gateway administration |
| reconcile tick → running container | An automated rotation recreates a live, funded service without asking |
| node logs → operator and anyone who can read them | The rotation announcement crosses this boundary |

## STRIDE Threat Register

| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-01-72 | Elevation of Privilege | a node continuing to answer to the published default after the code fix ships | critical | mitigate | Detection and rotation run on the same reconcile tick that already materialises secrets, so an affected node heals without operator action; step 6 proves the old credential is rejected afterwards |
| T-01-73 | Denial of Service | rotation recreating the gateway repeatedly, or in a loop, on every tick | high | mitigate | Rotation is denylist-exact and therefore self-terminating — the rotated value is not on the denylist, so the next tick is a no-op; an idempotence test and step 3's log observation both cover it |
| T-01-74 | Information Disclosure | the new credential appearing in a log line, status output or deploy transcript | high | mitigate | An explicit prohibition, the log line is specified to name a path rather than a value, and step 3 requires confirming no value appears in the log |
| T-01-75 | Tampering | a hand-rolled remove-and-recreate losing the gateway's data directory | critical | mitigate | The action forbids hand-rolled teardown, routes the recreate through the existing `secret_env_hash` path, and an acceptance criterion greps the diff for teardown primitives; step 5 verifies the data on the node |
| T-01-76 | Repudiation | signing off without ever exercising the rotation because the node happened to be clean | high | mitigate | Step 1 forces the executor to declare which case they are in, and the planner assumption requires deliberately seeding the affected state if the node is clean |
| T-01-77 | Denial of Service | an operator locked out of their own gateway by a rotation they cannot recover from | high | mitigate | Task 1 requires the recovery path to be settled and named in the log line before this plan is done; step 6 proves the new credential actually authenticates |
| T-01-SC | Tampering | npm/pip/cargo installs | high | mitigate | This plan installs nothing — two Rust edits. If an implementation choice would add a crate, stop: RESEARCH.md's Package Legitimacy Audit must cover it first, with a blocking human checkpoint for any `[ASSUMED]`/`[SUS]` entry |
</threat_model>

<verification>
- `cd core && cargo test -p archipelago` — green.
- The blocking checkpoint's eight steps, run on archi-dev-box, with the affected-or-clean case declared.
- `tests/lifecycle/run-gate.sh` green on-node after the change.
</verification>

<success_criteria>
- An affected node rotates itself once, keeps its data, ports and container name, and comes back healthy.
- The old shipped credential no longer authenticates; the new one does.
- A fresh install produces a different credential again.
- The rotation is announced without ever printing the value, and the operator has a named way to retrieve it.
- The second dev-pair node's absence is recorded as a gap rather than glossed over.
</success_criteria>

<output>
Create `.planning/phases/01-federation-mesh-hardening/01-16-SUMMARY.md` when done, recording the
affected-or-clean verdict for archi-dev-box, the deploy command used, the gate result, the operator
recovery path, and any issue text verbatim.
Stage by explicit path, commit, and `git push gitea-ai main`.
</output>