lfg2025/archy
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases 44 Wiki Activity
archy/docs/hotfix-process.md
archipelago 1394994e0c docs(ops): fix runbook/hotfix/troubleshooting against reality + redact committed sudo password 
- operations-runbook: redact the plaintext sudo password (rotate it — it
  stays in git history); replace the nonexistent test-cross-node.sh /
  test-reboot-survival.sh with the real entry points (lifecycle gate,
  multinode smoke, e2e/post-install scripts)
- hotfix-process: de-hardcode the v1.0.x framing, point releases at
  create-release.sh + primary Gitea, correct the rollback backup paths
  (/opt/archipelago/rollback + updater backup dir, not /usr/local/bin)
- troubleshooting: connectivity probe start9.com -> debian.org
- INSTALL.sh: drop removed endurain app, point at developer-guide.md
  (development-setup.md never existed)
- tests/lifecycle/TESTING.md: replace the stale 2026-06-21 mid-session
  resume block with a resolved note; release-gate item 8 reflects the
  2026-07-08 version decision (1.8.0-alpha)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 18:46:51 -04:00

1.9 KiB
Raw Blame History

Hotfix Process

For critical bugs discovered after a tagged release.

Severity Classification

Level Response Time Examples
P0 — Critical < 4 hours Data loss, security vulnerability, node bricked
P1 — High < 24 hours App won't start, auth broken, major UI failure
P2 — Medium < 72 hours Non-critical feature broken, performance regression
P3 — Low Next release Cosmetic, minor UX, edge cases

Hotfix Workflow

1. Triage

  • Reproduce the issue on dev server (192.168.1.228)
  • Classify severity (P0-P3)
  • P0/P1: proceed immediately. P2/P3: add to the next release (docs/UNIFIED-TASK-TRACKER.md).

2. Fix

  • Create branch: hotfix/vX.Y.Z-description
  • Fix the issue with minimal code changes
  • Run full test suite: cd neode-ui && npm test && npm run type-check
  • Deploy to dev server: ./scripts/deploy-to-target.sh --live
  • Verify fix on live server

3. Release

  • Merge hotfix branch to main
  • Tag: vX.Y.Z (increment patch version)
  • Cut the release with ./scripts/create-release.sh X.Y.Z (updates releases/manifest.json and signs it)
  • Push main + tags to the primary Gitea release server so nodes pick it up OTA

4. Communicate

  • Update RELEASE-NOTES with hotfix details
  • Note in CHANGELOG.md

Monitoring Dashboards

  • Uptime monitor: /var/lib/archipelago/uptime-monitor/summary.json
  • Soak test: /tmp/stability-test-*.log on dev server
  • Health endpoint: http://192.168.1.228/health

Rollback

If a hotfix causes regressions:

  1. The updater self-verifies after applying (health check on restart) and rolls the binary back automatically if the new one fails to come up
  2. Point releases/manifest.json back at the last-known-good version and push
  3. Backend binary backups: /opt/archipelago/rollback/archipelago.bak (deploy script) and /var/lib/archipelago/update-backup/archipelago.bak (self-update.sh)