archy/.planning/RESUME-2026-08-05-appgate-fixes.md

Resume — 2026-08-05 (app gate, releases .122–.125)

Paste the block at the bottom into a new session.

Where things stand

• v1.7.124-alpha is SHIPPED (signed with the NEW root, published, verified).
• Signed catalog is LIVE carrying two hotfixes made after .124: the repaired bitcoin start script and the fedimint 8175 removal. Last commit: 4ace62fa.
• Release-root rotation is COMPLETE. .122 was the last release signed with the old key; .123/.124 and all catalogs use the new one. No override needed.

Two bugs I introduced in .124 (both fixed, both instructive)

1. Bitcoin vanished from every node. I put a # comment INSIDE the manifest's folded YAML scalar (>-), where # is not a comment — it reaches the shell, and folding joins lines with spaces so it commented out the if ... then while the more-indented echo survived, leaving an orphan fi. Container exited instantly; app detection is container-based so the app disappeared. Guard added: scripts/check-manifest-shell.py runs sh -n over every embedded manifest script and rejects # in these scalars; wired into tests/release/run.sh.
2. Fedimint crash-looped. I declared port 8175 on the fedimint app so the gate could name it — but 8175 is served by the separate archy-fedimint-ui companion. The orchestrator then tried to publish 8175 from fedimintd, collided, and start_container failed forever. Removed. Rule: never declare a port on an app whose container does not actually serve it.

Also: I published an UNSIGNED catalog at one point, which nodes correctly reject — they silently keep their old cached copy. Always verify 'signature' in catalog on the live URL after publishing.

OPEN TASKS

1. indeedhub crash-loop — NOT mine, needs a real fix. indeedhub-minio is absent on .38 and .88, so nginx fails with host not found in upstream "minio" and both indeedhub and indeedhub-api exit(1). The stack member never gets created. Look at api/rpc/package/stacks.rs + dependencies.rs.
2. Verify .38 refetched the signed catalog and bitcoin-knots starts. .88 already did (signed: True, script fixed).
3. Deploy the .125 build to archi-dev-box for operator confirmation. Binary is built at core/target/release/archipelago with: app-login page using the sidebar A mark (favico-black-v2.svg) not the wordmark; page pinned to 100svh + position:fixed so mobile stays centred and the keyboard overlays instead of scrolling; install-version modal icon uses object-contain so non-square icons are not cropped. Operator has not seen these yet.
4. Cut v1.7.125-alpha once confirmed. Sign with the NEW mnemonic.

Traps that cost time today

• create-release.sh says "sign, then re-run" — re-running regenerates the manifest and DESTROYS the signature, and its clean-tree check blocks anyway. Do steps 7/8 by hand: git add version+changelog+manifest → commit chore: release vX → git tag -a vX → push main → push the tag explicitly → git ls-remote --tags to prove it → publish-release-assets.sh.
• The release gate's cargo-test-weekly times out on the compile after any version bump. Pre-warm: CARGO_INCREMENTAL=0 cargo test --manifest-path core/Cargo.toml -p archipelago --no-run.
• The frontend version check fails until the in-app What's New block for that version exists (neode-ui/src/views/settings/AccountInfoSection.vue) — that string is what it greps for.
• generate-app-catalog.py writes APP_LAUNCH_PORTS one-per-line; rustfmt packs it, so run cargo fmt after any catalog sync or the gate fails.
• Manifest changes reach nodes via the SIGNED CATALOG, not the binary. A manifest hotfix needs only a catalog re-sign — no release.

Fleet

SSH: sshpass -p 'ThisIsWeb54321!' ssh archipelago@<ip> (note the !; @ is older and still works on some). RPC/node password differs per node — the ! one failed RPC login on .38.

• 100.69.68.39 archi-dev-box — dev target
• 100.82.34.38 archipelago-1
• 100.70.96.88 austin-sapien
• 100.64.204.114 .228 shorty-s — in real use, treat carefully

Force a catalog refresh on a node: Settings → App Updates → Check for updates, or sudo rm -f /var/lib/archipelago/app-catalog.json && sudo systemctl restart archipelago.

All fleet nodes were repaired from Restart=on-failure → Restart=always; a node with the old value stays DEAD after an in-process update (the updater exits cleanly and systemd reads that as success). bootstrap::ensure_restart_policy() now self-heals it.

---

PASTE THIS INTO THE NEW SESSION

Resume the archy work from 2026-08-05. Read .planning/RESUME-2026-08-05-appgate-fixes.md and the memory notes project_fleet_ota_restart_policy_incident and project_v1_7_121_shipped_appgate first.

v1.7.124-alpha is shipped and the signed catalog is live with two hotfixes (bitcoin start script, fedimint 8175). Four things are open, in order:

1. Fix the indeedhub crash-loop: indeedhub-minio is absent on .38 and .88 so nginx fails on upstream "minio" and indeedhub + indeedhub-api exit(1). This one is pre-existing, not from the port work.
2. Verify .38 refetched the signed catalog and bitcoin-knots starts (.88 already did).
3. Deploy the built .125 binary + frontend to archi-dev-box (100.69.68.39) so I can confirm the app-login page (A mark, mobile centring, keyboard behaviour) and the install-modal icon.
4. Then cut v1.7.125-alpha — I sign with the new mnemonic.

Do not re-run create-release.sh after signing; it destroys the signature — do the commit/tag/publish steps by hand as the resume doc describes.