1394994e0c
- operations-runbook: redact the plaintext sudo password (rotate it — it stays in git history); replace the nonexistent test-cross-node.sh / test-reboot-survival.sh with the real entry points (lifecycle gate, multinode smoke, e2e/post-install scripts) - hotfix-process: de-hardcode the v1.0.x framing, point releases at create-release.sh + primary Gitea, correct the rollback backup paths (/opt/archipelago/rollback + updater backup dir, not /usr/local/bin) - troubleshooting: connectivity probe start9.com -> debian.org - INSTALL.sh: drop removed endurain app, point at developer-guide.md (development-setup.md never existed) - tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session resume block with a resolved note; release-gate item 8 reflects the 2026-07-08 version decision (1.8.0-alpha) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
53 lines
1.9 KiB
Markdown
53 lines
1.9 KiB
Markdown
# Hotfix Process
|
|
|
|
For critical bugs discovered after a tagged release.
|
|
|
|
## Severity Classification
|
|
|
|
| Level | Response Time | Examples |
|
|
|-------|--------------|---------|
|
|
| P0 — Critical | < 4 hours | Data loss, security vulnerability, node bricked |
|
|
| P1 — High | < 24 hours | App won't start, auth broken, major UI failure |
|
|
| P2 — Medium | < 72 hours | Non-critical feature broken, performance regression |
|
|
| P3 — Low | Next release | Cosmetic, minor UX, edge cases |
|
|
|
|
## Hotfix Workflow
|
|
|
|
### 1. Triage
|
|
- Reproduce the issue on dev server (192.168.1.228)
|
|
- Classify severity (P0-P3)
|
|
- P0/P1: proceed immediately. P2/P3: add to the next release (`docs/UNIFIED-TASK-TRACKER.md`).
|
|
|
|
### 2. Fix
|
|
- Create branch: `hotfix/vX.Y.Z-description`
|
|
- Fix the issue with minimal code changes
|
|
- Run full test suite: `cd neode-ui && npm test && npm run type-check`
|
|
- Deploy to dev server: `./scripts/deploy-to-target.sh --live`
|
|
- Verify fix on live server
|
|
|
|
### 3. Release
|
|
- Merge hotfix branch to `main`
|
|
- Tag: `vX.Y.Z` (increment patch version)
|
|
- Cut the release with `./scripts/create-release.sh X.Y.Z` (updates
|
|
`releases/manifest.json` and signs it)
|
|
- Push `main` + tags to the primary Gitea release server so nodes pick it up OTA
|
|
|
|
### 4. Communicate
|
|
- Update RELEASE-NOTES with hotfix details
|
|
- Note in CHANGELOG.md
|
|
|
|
## Monitoring Dashboards
|
|
|
|
- **Uptime monitor**: `/var/lib/archipelago/uptime-monitor/summary.json`
|
|
- **Soak test**: `/tmp/stability-test-*.log` on dev server
|
|
- **Health endpoint**: `http://192.168.1.228/health`
|
|
|
|
## Rollback
|
|
|
|
If a hotfix causes regressions:
|
|
1. The updater self-verifies after applying (health check on restart) and rolls the
|
|
binary back automatically if the new one fails to come up
|
|
2. Point `releases/manifest.json` back at the last-known-good version and push
|
|
3. Backend binary backups: `/opt/archipelago/rollback/archipelago.bak` (deploy script)
|
|
and `/var/lib/archipelago/update-backup/archipelago.bak` (`self-update.sh`)
|