---
created: 2026-08-01T09:45:00.000Z
title: Verify FED-07 gateway credential rotation on a real node (01-16 Task 2)
area: security
severity: major
files:
  - core/archipelago/src/container/secrets.rs
  - core/archipelago/src/container/prod_orchestrator.rs
  - .planning/phases/01-federation-mesh-hardening/01-16-PLAN.md
---

## Problem

FED-07's code is complete and pushed (`42652547` removed every shipped credential,
`9e2d2ef2` added detection + rotation for nodes already carrying one), but **plan 01-16's
blocking Task 2 checkpoint has never been run**, so the requirement is not closed. Two
things are unproven:

1. **That the recreate actually preserves the gateway's data.** Rotation deliberately
   avoids any teardown: it changes the credential, which changes `secret_env_hash`, which
   the drift check reads as a container-label mismatch, which fires the platform's own
   recreate around the unchanged data directory, ports, volumes and container name. That
   chain is confirmed by reading the code (`prod_orchestrator.rs:3309` computes the hash,
   `:3374` compares the label) but has **never been observed running**.
2. **That the rotated credential actually authenticates** and the old shipped one is
   rejected.

## Why it was deferred (2026-08-01)

Checkpoint step 1 was run read-only on archi-dev-box. Findings:

- **The node is CLEAN** — the hash file is present, `600`, `archipelago:archipelago`, and
  is *not* the shipped default. The legacy `fedimint-gateway-password` file also exists.
- **There is no `fedimint-gateway` container.** The app is installed at
  `/opt/archipelago/apps/fedimint-gateway`, but nothing is running and
  `/var/lib/archipelago/fedimint-gateway` is empty. `archy-fedimint-ui` and
  `fedimint-clientd` are running, but they are not the gateway.

So the rotation path cannot fire naturally here, and steps 5–6 (data survived, gateway
authenticates) have nothing to exercise. Proving it on this box means installing and
starting the gateway first, then deliberately seeding the old value — which the plan's
Planner Assumption already anticipates and requires be recorded.

The deploy itself is the blocker: **30 containers are running with 4–8 days uptime**
(IndeeHub relay/minio/postgres/ffmpeg, Immich, BTCPay + nbxplorer, netbird, portainer,
strfry, searxng, the archy UI set), the `archipelago` system service is active, and
CLAUDE.md's standing rule is that restarting it SIGKILLs containers until Quadlet is the
default. Step 8 then wants `tests/lifecycle/run-gate.sh` on the same box, which cycles
install/stop/start/reinstall/reboot-survive. Dorian chose to hold rather than take that
blast radius unattended.

## Solution

Run 01-16 Task 2's eight steps in a window where restarting `archipelago` on the target is
acceptable. Two viable routes:

- **On archi-dev-box:** install + start `fedimint-gateway`, put some state in its data
  directory, seed the old hash into
  `/var/lib/archipelago/secrets/fedimint-gateway-hash`, deploy this build, restart, then
  run steps 3–8. Record that the affected state was seeded deliberately.
- **On a node that runs the gateway for real** (.228 is the heavy 14-app node): the
  rotation path may exist naturally there, which is stronger evidence than seeding.

Either way, record in `01-16-SUMMARY.md`: the affected-or-clean verdict, the exact deploy
command, whether the `secret_env_hash` recreate was observed firing, the gate result, and
the fact that archy-x250-dev was offline (single-node verification with the second-node
gap recorded honestly is the expected pattern).

## Related, found while doing 01-16

- **Operator recovery gap:** `handle_package_credentials`
  (`core/archipelago/src/api/rpc/package/install.rs:2093`) is a hardcoded if-chain covering
  only `filebrowser` and `photoprism`. `fedimint-gateway` falls through to an empty list,
  so after a rotation the operator has no in-UI way to get the new password — only the
  `0600` file the log line names. Adding one arm closes it; the UI already renders whatever
  the RPC returns.
- **`apps/photoprism/manifest.yml:35` ships `PHOTOPRISM_ADMIN_PASSWORD=archipelago`**, and
  that same literal is handed to the UI. Same defect class as FED-07 on a different app —
  wants its own requirement.
</content>