archy/scripts/run-tests.sh

88 lines
2.1 KiB
Bash
Raw Normal View History

release(v1.7.41-alpha): post-OTA auto-rollback so a bad release cannot strand the fleet Closes failure mode FM5 from docs/bulletproof-containers.md: the v1.7.38 + v1.7.39 rollouts left every affected node on an unreachable UI (nginx 500) with no recovery path short of SSH. This release adds a self-check guardrail to the update flow. What changed: - apply_update() writes a pending-verify marker with old+new version and a 150s deadline immediately before scheduling the service restart. - verify_pending_update() runs from main.rs startup. If the marker is present and within its freshness window, the new binary waits 15s for nginx + backend to settle, then probes https://127.0.0.1/ every 5s for up to 90s (self-signed certs accepted). - On any probe success within the window, the marker is cleared and nothing else happens. - On window-exhaust, the new binary: 1. Moves the broken /opt/archipelago/web-ui to web-ui.failed.<ts> (quarantined, not deleted, so we can post-mortem). 2. Restores web-ui.bak on top of web-ui. 3. Calls rollback_update() to restore the previous binary. 4. Updates state.current_version to reflect the rollback. 5. systemctl --no-block restart archipelago so the OLD binary boots. - Markers older than 10 minutes are treated as stale and cleared without probing, so a crashed-during-startup marker from weeks ago cannot spontaneously roll back a healthy node on a later reboot. - rollback_update() binary copy now goes through host_sudo instead of tokio::fs::copy, so it escapes the service's ProtectSystem=strict mount namespace. Without this, the rollback silently failed with EROFS on /usr/local/bin and orphaned the rollback - the exact opposite of what auto-rollback is for. Tests: 4 new unit tests in update::tests covering marker round-trip, absent-marker noop, no-panic on verify_pending_update with nothing to verify, and an invariant assert that the 90s probe window stays below the 600s stale threshold. All passing. Side fix: scripts/create-release-manifest.sh was dying with exit 141 (SIGPIPE from tar tvzf pipe head pipe awk) under set -euo pipefail. Replaced with a single awk NR==1 that doesn't short-circuit the upstream pipe, so the release-build flow is idempotent again.
2026-04-22 16:14:35 -04:00
#!/usr/bin/env bash
#
# Run all Archipelago tests: frontend (local) + backend (dev server via SSH).
# Exit 0 only if both pass.
#
set -euo pipefail
SSH_KEY="${ARCHIPELAGO_SSH_KEY:-$HOME/.ssh/archipelago-deploy}"
SSH_HOST="${ARCHIPELAGO_SSH_HOST:-archipelago@192.168.1.228}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
FRONTEND_OK=0
BACKEND_OK=0
echo "========================================="
echo " Archipelago Test Runner"
echo "========================================="
echo ""
# --- Frontend Tests ---
echo "--- Frontend Tests (local) ---"
if (cd "$PROJECT_DIR/neode-ui" && npm test 2>&1); then
echo "✅ Frontend tests PASSED"
FRONTEND_OK=1
else
echo "❌ Frontend tests FAILED"
fi
echo ""
# --- Backend Tests (on dev server) ---
echo "--- Backend Tests (dev server) ---"
# Sync source to server
echo "Syncing source to dev server..."
rsync -az --exclude 'target' --exclude 'node_modules' --exclude '.git' \
-e "ssh -i $SSH_KEY" \
"$PROJECT_DIR/core/" "$SSH_HOST:~/archy/core/" 2>&1
# Run tests on server
if ssh -i "$SSH_KEY" "$SSH_HOST" \
"source ~/.cargo/env && cd ~/archy/core && cargo test -p archipelago 2>&1"; then
echo "✅ Backend unit tests PASSED"
BACKEND_OK=1
else
echo "❌ Backend unit tests FAILED"
fi
echo ""
# --- Integration Tests ---
echo "--- Integration Tests (dev server) ---"
if ssh -i "$SSH_KEY" "$SSH_HOST" \
"source ~/.cargo/env && cd ~/archy/core && cargo test --test rpc_integration 2>&1"; then
echo "✅ Integration tests PASSED"
else
echo "❌ Integration tests FAILED"
BACKEND_OK=0
fi
echo ""
echo "========================================="
echo " Results"
echo "========================================="
if [ "$FRONTEND_OK" -eq 1 ]; then
echo " Frontend: ✅ PASS"
else
echo " Frontend: ❌ FAIL"
fi
if [ "$BACKEND_OK" -eq 1 ]; then
echo " Backend: ✅ PASS"
else
echo " Backend: ❌ FAIL"
fi
echo "========================================="
if [ "$FRONTEND_OK" -eq 1 ] && [ "$BACKEND_OK" -eq 1 ]; then
echo "All tests passed!"
exit 0
else
echo "Some tests failed."
exit 1
fi