archy/.planning/RESUME-2026-08-05-appgate-fixes.md

# Resume — 2026-08-05 (app gate, releases .122–.125)

Paste the block at the bottom into a new session.

## Where things stand

- **v1.7.124-alpha is SHIPPED** (signed with the NEW root, published, verified).
- **Signed catalog is LIVE** carrying two hotfixes made after .124:
  the repaired bitcoin start script and the fedimint 8175 removal.
  Last commit: `4ace62fa`.
- **Release-root rotation is COMPLETE.** .122 was the last release signed with
  the old key; .123/.124 and all catalogs use the new one. No override needed.

## Two bugs I introduced in .124 (both fixed, both instructive)

1. **Bitcoin vanished from every node.** I put a `#` comment INSIDE the
   manifest's folded YAML scalar (`>-`), where `#` is not a comment — it
   reaches the shell, and folding joins lines with spaces so it commented out
   the `if ... then` while the more-indented `echo` survived, leaving an orphan
   `fi`. Container exited instantly; app detection is container-based so the
   app disappeared. **Guard added:** `scripts/check-manifest-shell.py` runs
   `sh -n` over every embedded manifest script and rejects `#` in these
   scalars; wired into `tests/release/run.sh`.
2. **Fedimint crash-looped.** I declared port 8175 on the `fedimint` app so the
   gate could name it — but 8175 is served by the separate `archy-fedimint-ui`
   companion. The orchestrator then tried to publish 8175 from fedimintd,
   collided, and `start_container` failed forever. Removed. **Rule: never
   declare a port on an app whose container does not actually serve it.**

Also: I published an UNSIGNED catalog at one point, which nodes correctly
reject — they silently keep their old cached copy. **Always verify
`'signature' in catalog` on the live URL after publishing.**

## OPEN TASKS

1. **indeedhub crash-loop — NOT mine, needs a real fix.** `indeedhub-minio` is
   **absent** on `.38` and `.88`, so nginx fails with
   `host not found in upstream "minio"` and both `indeedhub` and
   `indeedhub-api` exit(1). The stack member never gets created. Look at
   `api/rpc/package/stacks.rs` + `dependencies.rs`.
2. **Verify `.38` refetched the signed catalog** and bitcoin-knots starts.
   `.88` already did (signed: True, script fixed).
3. **Deploy the .125 build to archi-dev-box for operator confirmation.**
   Binary is built at `core/target/release/archipelago` with: app-login page
   using the sidebar **A mark** (`favico-black-v2.svg`) not the wordmark;
   page pinned to `100svh` + `position:fixed` so mobile stays centred and the
   keyboard overlays instead of scrolling; install-version modal icon uses
   `object-contain` so non-square icons are not cropped. **Operator has not
   seen these yet.**
4. **Cut v1.7.125-alpha** once confirmed. Sign with the **NEW** mnemonic.

## Traps that cost time today

- `create-release.sh` says "sign, then re-run" — **re-running regenerates the
  manifest and DESTROYS the signature**, and its clean-tree check blocks
  anyway. Do steps 7/8 by hand: `git add` version+changelog+manifest →
  commit `chore: release vX` → `git tag -a vX` → push main → **push the tag
  explicitly** → `git ls-remote --tags` to prove it → `publish-release-assets.sh`.
- The release gate's `cargo-test-weekly` times out on the **compile** after any
  version bump. Pre-warm: `CARGO_INCREMENTAL=0 cargo test --manifest-path
  core/Cargo.toml -p archipelago --no-run`.
- The frontend version check fails until the in-app **What's New** block for
  that version exists (`neode-ui/src/views/settings/AccountInfoSection.vue`) —
  that string is what it greps for.
- `generate-app-catalog.py` writes `APP_LAUNCH_PORTS` one-per-line; rustfmt
  packs it, so run `cargo fmt` after any catalog sync or the gate fails.
- **Manifest changes reach nodes via the SIGNED CATALOG, not the binary.** A
  manifest hotfix needs only a catalog re-sign — no release.

## Fleet

SSH: `sshpass -p 'ThisIsWeb54321!' ssh archipelago@<ip>` (note the `!`; `@`
is older and still works on some). RPC/node password differs per node — the
`!` one failed RPC login on `.38`.

- `100.69.68.39` archi-dev-box — dev target
- `100.82.34.38` archipelago-1
- `100.70.96.88` austin-sapien
- `100.64.204.114` .228 shorty-s — **in real use, treat carefully**

**Force a catalog refresh on a node:** Settings → App Updates → Check for
updates, or `sudo rm -f /var/lib/archipelago/app-catalog.json && sudo
systemctl restart archipelago`.

**All fleet nodes were repaired** from `Restart=on-failure` →
`Restart=always`; a node with the old value stays DEAD after an in-process
update (the updater exits cleanly and systemd reads that as success).
`bootstrap::ensure_restart_policy()` now self-heals it.

---

## PASTE THIS INTO THE NEW SESSION

Resume the archy work from 2026-08-05. Read
`.planning/RESUME-2026-08-05-appgate-fixes.md` and the memory notes
`project_fleet_ota_restart_policy_incident` and
`project_v1_7_121_shipped_appgate` first.

v1.7.124-alpha is shipped and the signed catalog is live with two hotfixes
(bitcoin start script, fedimint 8175). Four things are open, in order:

1. Fix the indeedhub crash-loop: `indeedhub-minio` is absent on .38 and .88 so
   nginx fails on upstream "minio" and indeedhub + indeedhub-api exit(1). This
   one is pre-existing, not from the port work.
2. Verify .38 refetched the signed catalog and bitcoin-knots starts (.88
   already did).
3. Deploy the built .125 binary + frontend to archi-dev-box (100.69.68.39) so
   I can confirm the app-login page (A mark, mobile centring, keyboard
   behaviour) and the install-modal icon.
4. Then cut v1.7.125-alpha — I sign with the new mnemonic.

Do not re-run create-release.sh after signing; it destroys the signature —
do the commit/tag/publish steps by hand as the resume doc describes.