feat(update): auto-rollback OTA binaries that crash-loop before they can self-verify
.198 crash-looped 236x on a truncated OTA binary with a good backup sitting in update-backup/ — verify_pending_update() can't help when the new binary never runs. New scripts/ota-crash-guard.sh runs as root from ExecStartPre: while the post-OTA pending-verify marker exists it counts start attempts, and after 5 failures restores the backup binary (atomic rename), replaces the marker with an update-rolled-back.json tombstone, and lets the service come back on the previous version. Wiring: fresh ISOs get the ExecStartPre line in the unit file; existing nodes get a drop-in installed by apply_update's runtime component step on their next OTA. '+-' prefix so a missing or failing guard can never block the service. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
837cfdfd1f
commit
25162ee846
@@ -25,6 +25,11 @@ ExecStartPre=+/bin/bash -c 'mkdir -p /run/user/1000 /var/lib/containers && chown
|
||||
# once a VPN/bridge interface exists (netbird's wg tunnel sorted first and
|
||||
# poisoned every host_ip consumer). Falls back to hostname -I when routeless.
|
||||
ExecStartPre=+/bin/bash -c 'mkdir -p /var/lib/archipelago && chown archipelago:archipelago /var/lib/archipelago && IP=$(ip -4 route show default 2>/dev/null | sed -n "s/.* src \([0-9.]*\).*/\1/p" | head -1); [ -n "$$IP" ] || IP=$(hostname -I 2>/dev/null | awk "{print $$1}"); echo "ARCHIPELAGO_HOST_IP=$$IP" > /var/lib/archipelago/host-ip.env && chown archipelago:archipelago /var/lib/archipelago/host-ip.env'
|
||||
# OTA crash-loop guard: if a just-applied binary can't start (SEGV loop), the
|
||||
# in-binary post-OTA probe never runs — this restores the update-backup binary
|
||||
# after 5 failed start attempts while the pending-verify marker exists.
|
||||
# "-" so a missing/failed guard can never block the service itself.
|
||||
ExecStartPre=+-/opt/archipelago/scripts/ota-crash-guard.sh
|
||||
ExecStart=/usr/local/bin/archipelago
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
Reference in New Issue
Block a user