archy/.planning/todos/pending/2026-08-01-verify-fed07-gateway-credential-rotation-on-node.md

created, title, area, severity, files

created	title	area	severity	files
2026-08-01T09:45:00.000Z	Verify FED-07 gateway credential rotation on a real node (01-16 Task 2)	security	major	core/archipelago/src/container/secrets.rs core/archipelago/src/container/prod_orchestrator.rs .planning/phases/01-federation-mesh-hardening/01-16-PLAN.md

Problem

FED-07's code is complete and pushed (42652547 removed every shipped credential, 9e2d2ef2 added detection + rotation for nodes already carrying one), but plan 01-16's blocking Task 2 checkpoint has never been run, so the requirement is not closed. Two things are unproven:

1. That the recreate actually preserves the gateway's data. Rotation deliberately avoids any teardown: it changes the credential, which changes secret_env_hash, which the drift check reads as a container-label mismatch, which fires the platform's own recreate around the unchanged data directory, ports, volumes and container name. That chain is confirmed by reading the code (prod_orchestrator.rs:3309 computes the hash, :3374 compares the label) but has never been observed running.
2. That the rotated credential actually authenticates and the old shipped one is rejected.

Why it was deferred (2026-08-01)

Checkpoint step 1 was run read-only on archi-dev-box. Findings:

• The node is CLEAN — the hash file is present, 600, archipelago:archipelago, and is not the shipped default. The legacy fedimint-gateway-password file also exists.
• There is no fedimint-gateway container. The app is installed at /opt/archipelago/apps/fedimint-gateway, but nothing is running and /var/lib/archipelago/fedimint-gateway is empty. archy-fedimint-ui and fedimint-clientd are running, but they are not the gateway.

So the rotation path cannot fire naturally here, and steps 5–6 (data survived, gateway authenticates) have nothing to exercise. Proving it on this box means installing and starting the gateway first, then deliberately seeding the old value — which the plan's Planner Assumption already anticipates and requires be recorded.

The deploy itself is the blocker: 30 containers are running with 4–8 days uptime (IndeeHub relay/minio/postgres/ffmpeg, Immich, BTCPay + nbxplorer, netbird, portainer, strfry, searxng, the archy UI set), the archipelago system service is active, and CLAUDE.md's standing rule is that restarting it SIGKILLs containers until Quadlet is the default. Step 8 then wants tests/lifecycle/run-gate.sh on the same box, which cycles install/stop/start/reinstall/reboot-survive. Dorian chose to hold rather than take that blast radius unattended.

Solution

Run 01-16 Task 2's eight steps in a window where restarting archipelago on the target is acceptable. Two viable routes:

• On archi-dev-box: install + start fedimint-gateway, put some state in its data directory, seed the old hash into /var/lib/archipelago/secrets/fedimint-gateway-hash, deploy this build, restart, then run steps 3–8. Record that the affected state was seeded deliberately.
• On a node that runs the gateway for real (.228 is the heavy 14-app node): the rotation path may exist naturally there, which is stronger evidence than seeding.

Either way, record in 01-16-SUMMARY.md: the affected-or-clean verdict, the exact deploy command, whether the secret_env_hash recreate was observed firing, the gate result, and the fact that archy-x250-dev was offline (single-node verification with the second-node gap recorded honestly is the expected pattern).

Related, found while doing 01-16

• Operator recovery gap: handle_package_credentials (core/archipelago/src/api/rpc/package/install.rs:2093) is a hardcoded if-chain covering only filebrowser and photoprism. fedimint-gateway falls through to an empty list, so after a rotation the operator has no in-UI way to get the new password — only the 0600 file the log line names. Adding one arm closes it; the UI already renders whatever the RPC returns.
• apps/photoprism/manifest.yml:35 ships PHOTOPRISM_ADMIN_PASSWORD=archipelago, and that same literal is handed to the UI. Same defect class as FED-07 on a different app — wants its own requirement.