docs(ops): fix runbook/hotfix/troubleshooting against reality + redact committed sudo password
- operations-runbook: redact the plaintext sudo password (rotate it — it stays in git history); replace the nonexistent test-cross-node.sh / test-reboot-survival.sh with the real entry points (lifecycle gate, multinode smoke, e2e/post-install scripts) - hotfix-process: de-hardcode the v1.0.x framing, point releases at create-release.sh + primary Gitea, correct the rollback backup paths (/opt/archipelago/rollback + updater backup dir, not /usr/local/bin) - troubleshooting: connectivity probe start9.com -> debian.org - INSTALL.sh: drop removed endurain app, point at developer-guide.md (development-setup.md never existed) - tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session resume block with a resolved note; release-gate item 8 reflects the 2026-07-08 version decision (1.8.0-alpha) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
parent
9ce8a0df05
commit
1394994e0c
@ -122,7 +122,7 @@ echo ""
|
||||
# Install custom app dependencies
|
||||
echo "Installing custom app dependencies..."
|
||||
|
||||
for app in did-wallet endurain morphos-server router; do
|
||||
for app in did-wallet morphos-server router; do
|
||||
if [ -d "apps/$app" ]; then
|
||||
echo " - Installing $app dependencies..."
|
||||
cd "apps/$app"
|
||||
@ -161,6 +161,6 @@ echo " http://localhost:8100"
|
||||
echo ""
|
||||
echo "For more information, see:"
|
||||
echo " - README.md"
|
||||
echo " - docs/development-setup.md"
|
||||
echo " - docs/developer-guide.md"
|
||||
echo " - apps/QUICKSTART.md"
|
||||
echo ""
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
# Hotfix Process
|
||||
|
||||
For critical bugs discovered after v1.0.0 release.
|
||||
For critical bugs discovered after a tagged release.
|
||||
|
||||
## Severity Classification
|
||||
|
||||
@ -16,10 +16,10 @@ For critical bugs discovered after v1.0.0 release.
|
||||
### 1. Triage
|
||||
- Reproduce the issue on dev server (192.168.1.228)
|
||||
- Classify severity (P0-P3)
|
||||
- P0/P1: proceed immediately. P2/P3: add to v1.1 roadmap.
|
||||
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
|
||||
|
||||
### 2. Fix
|
||||
- Create branch: `hotfix/v1.0.1-description`
|
||||
- Create branch: `hotfix/vX.Y.Z-description`
|
||||
- Fix the issue with minimal code changes
|
||||
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
|
||||
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
|
||||
@ -27,9 +27,10 @@ For critical bugs discovered after v1.0.0 release.
|
||||
|
||||
### 3. Release
|
||||
- Merge hotfix branch to `main`
|
||||
- Tag: `v1.0.1` (increment patch version)
|
||||
- Update release manifest for OTA updates (`releases/manifest.json`)
|
||||
- Push to both Gitea mirrors so nodes can pull via `self-update.sh`
|
||||
- Tag: `vX.Y.Z` (increment patch version)
|
||||
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
|
||||
`releases/manifest.json` and signs it)
|
||||
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
|
||||
|
||||
### 4. Communicate
|
||||
- Update RELEASE-NOTES with hotfix details
|
||||
@ -44,6 +45,8 @@ For critical bugs discovered after v1.0.0 release.
|
||||
## Rollback
|
||||
|
||||
If a hotfix causes regressions:
|
||||
1. OTA system supports rollback to previous version (`scripts/self-update.sh --rollback`)
|
||||
2. Point `releases/manifest.json` back at the last-known-good version and push to mirrors
|
||||
3. Backend binary backup at `/usr/local/bin/archipelago.bak`
|
||||
1. The updater self-verifies after applying (health check on restart) and rolls the
|
||||
binary back automatically if the new one fails to come up
|
||||
2. Point `releases/manifest.json` back at the last-known-good version and push
|
||||
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
|
||||
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
|
||||
|
||||
@ -5,7 +5,7 @@ Quick reference for common operational tasks on Archipelago nodes.
|
||||
**Primary node**: `192.168.1.228` (Arch 1)
|
||||
**Secondary node**: `192.168.1.198` (Arch 2)
|
||||
**SSH**: `ssh -i ~/.ssh/archipelago-deploy archipelago@{IP}`
|
||||
**Sudo**: `echo 'EwPDR8q45l0Upx@' | sudo -S {command}`
|
||||
**Sudo**: use the node's sudo password (kept out of this doc — never commit credentials)
|
||||
|
||||
---
|
||||
|
||||
@ -350,15 +350,17 @@ If a node responds to ping but SSH/HTTP are down:
|
||||
podman image prune -a
|
||||
```
|
||||
|
||||
## 17. Run Cross-Node Tests
|
||||
## 17. Run Tests
|
||||
|
||||
```bash
|
||||
# Full test suite (all features, 10 iterations)
|
||||
./scripts/test-cross-node.sh --iterations 10
|
||||
# Production lifecycle gate — run ON the node (uses local podman/systemctl):
|
||||
tests/lifecycle/run-gate.sh # see tests/lifecycle/TESTING.md
|
||||
ARCHY_ITERATIONS=5 tests/lifecycle/run-gate.sh
|
||||
|
||||
# Skip reboot tests
|
||||
./scripts/test-cross-node.sh --iterations 10 --skip-reboot
|
||||
# Cross-node suites (federation/mesh):
|
||||
tests/multinode/smoke.sh # see docs/multinode-testing-plan.md
|
||||
|
||||
# Reboot survival test (single node)
|
||||
./scripts/test-reboot-survival.sh --node 192.168.1.228 --iterations 3
|
||||
# E2E / post-install:
|
||||
./scripts/run-e2e-tests.sh
|
||||
./scripts/run-post-install-tests.sh
|
||||
```
|
||||
|
||||
@ -236,7 +236,7 @@ sudo journalctl -u archipelago -f
|
||||
**Diagnosis**:
|
||||
```bash
|
||||
# Check internet connectivity
|
||||
curl -s https://start9.com > /dev/null && echo "Internet OK" || echo "No internet"
|
||||
curl -s https://debian.org > /dev/null && echo "Internet OK" || echo "No internet"
|
||||
|
||||
# Check backend logs
|
||||
sudo journalctl -u archipelago --since "15 minutes ago" | grep -i update
|
||||
|
||||
@ -50,64 +50,15 @@ separate pass → `docs/multinode-testing-plan.md`.)
|
||||
`lnd` on .228 — they are synced and healthy; destructive cycles there would
|
||||
cost hours of resync.
|
||||
|
||||
### Session work log
|
||||
### Session work log (resolved)
|
||||
|
||||
| Date | App | Change | State |
|
||||
|---|---|---|---|
|
||||
| 2026-06-21 | fedimint-gateway / -clientd | **Generated-secrets system** (Pillar 4+5). New `generated_secrets:` manifest field (`hex16`/`hex32`/`bcrypt`); materialised generically at the `resolve_dynamic_env` chokepoint — atomic `0600`, rootless-owned, idempotent, and **self-healing** (recreates a wrongly `root:root`-owned secret via the service-owned dir, no chown/privilege). Removed per-app `ensure_fmcd_password` (−30 LoC). Fixes gateway never starting (`resolving secret_env` → missing/unreadable `fedimint-gateway-hash`). | ◐ code complete, `cargo check` + 3 unit tests green; **not yet deployed/validated on .228** |
|
||||
| 2026-06-21 | fedimint-gateway | Icon placeholder | ○ investigating: marketplace catalog has title+icon (fedimint.png, shared); `BUNDLED_APPS` frontend list omits fedimint → installed view falls back to 📦 |
|
||||
|
||||
### ⏯ RESUME POINT (2026-06-21, mid-session)
|
||||
|
||||
**Done (working tree, NOT git-committed):**
|
||||
- Generated-secrets system — all files below written, `cargo check` clean, 3 unit tests green.
|
||||
- Manifests declare `generated_secrets` (fmcd-password hex16; fedimint-gateway-hash bcrypt).
|
||||
- Tracker refreshed with 5 pillars + this log.
|
||||
|
||||
**In flight:**
|
||||
- Local release build RUNNING (`cd core && cargo build --release -p archipelago`,
|
||||
log `/tmp/archy-local-build.log`, output `core/target/release/archipelago`).
|
||||
⚠️ **.228 has NO cargo and NO rsync** — build LOCALLY on .116, ship binary + files
|
||||
via **tar-over-ssh** (`tar -cf - … | ssh … 'tar -xf -'`).
|
||||
|
||||
**Next steps (in order):**
|
||||
1. Wait for local build → `Finished`. scp/tar `core/target/release/archipelago` → .228.
|
||||
2. Ship updated manifests to **`/opt/archipelago/apps/fedimint-{gateway,clientd}/`** (canonical runtime dir; cwd-relative `apps` doesn't resolve — WorkingDirectory is empty).
|
||||
3. **Binary swap is SAFE for protected backends:** `archipelago.service` is
|
||||
`KillMode=control-group` BUT bitcoin-knots/electrumx/lnd conmons live under
|
||||
`user.slice/.../libpod-*.scope`, NOT the service cgroup. Only fedimint-clientd +
|
||||
immich conmons are in-cgroup (non-protected, reconciled back). `systemctl stop
|
||||
archipelago` → `cp` binary → `start`.
|
||||
4. Validate: install fedimint-gateway → assert `fedimint-gateway-hash` (0600,
|
||||
archipelago-owned) + `.pw` generated → container starts healthy.
|
||||
5. Run `tests/lifecycle/run-gate.sh` for the gateway (do NOT touch knots/electrumx/lnd).
|
||||
6. Frontend fixes (separate from binary): see icon/rename below; rebuild neode-ui,
|
||||
ship `dist + catalog.json + assets` to `/opt/archipelago/web-ui` (chown 1000:1000).
|
||||
|
||||
**Icon / naming (frontend, user-confirmed):**
|
||||
- Gateway icon = **reuse fedimint.png** (user choice). Static catalogs already map all 3
|
||||
→ fedimint.png; deployed `/catalog.json` on .228 also correct; `/api/app-catalog`
|
||||
(decoupled, dict form) returns no fedimint → frontend falls through to `/catalog.json`.
|
||||
Placeholder is therefore a **stale deployed bundle** and/or the **hardcoded fallback gap**:
|
||||
`getCuratedAppList()` in `neode-ui/src/views/discover/curatedApps.ts` omits
|
||||
fedimint-gateway + fedimint-clientd entirely — add both (icon fedimint.png).
|
||||
- Base **`fedimint` → display "Fedimint Guardian"** (user ask). Edit name/title in:
|
||||
`apps/fedimint/manifest.yml`, `app-catalog/catalog.json`,
|
||||
`neode-ui/public/catalog.json`, `web/dist/neode-ui/catalog.json`,
|
||||
`curatedApps.ts:101`. (`INSTALLED_ALIASES.fedimint = ['fedimint-gateway']` in curatedApps.ts.)
|
||||
|
||||
**.228 access:** `sshpass -p archipelago ssh archipelago@192.168.1.228`; UI/RPC pw
|
||||
`password123` (https). Binary `/usr/local/bin/archipelago` (v1.7.99-alpha).
|
||||
|
||||
### Generated-secrets — files touched
|
||||
|
||||
- `core/container/src/manifest.rs` — `GeneratedSecret` + `SecretGenKind` types, `ContainerConfig.generated_secrets`, validation (bare-filename, unique target files).
|
||||
- `core/container/src/lib.rs` — re-export the new types.
|
||||
- `core/archipelago/src/container/secrets.rs` — **new** generator module (atomic write, idempotent, self-heal) + 3 unit tests.
|
||||
- `core/archipelago/src/container/mod.rs` — register module.
|
||||
- `core/archipelago/src/container/prod_orchestrator.rs` — call `ensure_generated_secrets` in `resolve_dynamic_env`; drop fmcd special-case.
|
||||
- `core/archipelago/src/wallet/fedimint_client.rs` — delete orphaned `ensure_fmcd_password` (reader keeps `FMCD_PASSWORD_SECRET`).
|
||||
- `apps/fedimint-clientd/manifest.yml`, `apps/fedimint-gateway/manifest.yml` — declare `generated_secrets`.
|
||||
The 2026-06-21 mid-session resume block that lived here (generated-secrets
|
||||
rollout for fedimint-gateway/-clientd, icon/naming fixes) is **done and
|
||||
shipped**: the generated-secrets system is a platform primitive
|
||||
(`container.generated_secrets`, see `docs/app-manifest-spec.md`), the
|
||||
manifests declare it, and the single-node gate went green on .228 on
|
||||
2026-06-23. Day-to-day open items live in `docs/UNIFIED-TASK-TRACKER.md` —
|
||||
don't add session logs here.
|
||||
|
||||
---
|
||||
|
||||
@ -302,7 +253,7 @@ priority-ordered list of what's still open across these):
|
||||
7. ☑ Layman-readable changelog — `CHANGELOG.md` backfilled through v1.8.00-alpha
|
||||
(per `feedback_changelog_layman.md`)
|
||||
8. ☐ Tag pushed to origin + gitea-local + gitea-vps2 (per `feedback_ship_ritual.md`) —
|
||||
blocked on the version-naming decision, see unified tracker Tier 3
|
||||
version decided 2026-07-08 (`1.8.0-alpha`); tag once the pre-tag items above close
|
||||
|
||||
## How to update this document
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user