fix(tor): make the helper's restart actually recover Tor, and stop lying about it
The dashboard's "Restart Tor" button dispatched to this helper and always
got {"ok":true} back. Two real defects, in order of importance — and one
disproved theory, recorded so nobody re-chases it:
1. reset-failed was missing. Once tor@default fails enough times systemd
latches "Start request repeated too quickly" and refuses to start it at
all; a plain restart is then a no-op no matter which unit you name.
All three fleet nodes found dead on 2026-08-09 were in exactly that
state, which is why the button appeared to do nothing.
2. The result was unconditional. The write-torrc branch waited up to 30s
for SOCKS and then ignored the outcome; the restart branch slept 3s and
claimed success. The UI reported "restarted" over a dead daemon.
Disproved: this was NOT wrong-unit targeting. `systemctl restart tor`
does propagate to tor@default — measured on austin-sapien, MainPID
changed. tor@default is still addressed explicitly because it is the
unit carrying the failed state worth resetting.
restart_tor_daemon() now: reset-failed, restart tor@default (fall back to
tor on single-instance installs), wait up to 30s for SOCKS on 9050, and
return {"ok":false,"error":...} pointing at journalctl when it never
comes up. Callers may no longer report success without a live SOCKS port.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
2090cef3bf
commit
24e378c421
+62
-14
@@ -53,6 +53,54 @@ rm -f "$ACTION_FILE"
|
|||||||
|
|
||||||
ACTION_TYPE=$(echo "$ACTION" | python3 -c "import sys,json; print(json.load(sys.stdin).get('action',''))" 2>/dev/null || echo "")
|
ACTION_TYPE=$(echo "$ACTION" | python3 -c "import sys,json; print(json.load(sys.stdin).get('action',''))" 2>/dev/null || echo "")
|
||||||
|
|
||||||
|
# Restart the daemon that actually serves, and report whether it really came up.
|
||||||
|
#
|
||||||
|
# Why the dashboard's "Restart Tor" button did nothing on 2026-08-09, in order
|
||||||
|
# of importance — note it was NOT wrong-unit targeting: `systemctl restart tor`
|
||||||
|
# does propagate to tor@default (measured on austin-sapien, MainPID changed).
|
||||||
|
#
|
||||||
|
# 1. RESET-FAILED WAS MISSING. Once tor@default has failed enough times
|
||||||
|
# systemd latches "Start request repeated too quickly" and refuses to start
|
||||||
|
# it at all; a plain restart is then a no-op no matter which unit you name.
|
||||||
|
# All three broken nodes were in exactly that state.
|
||||||
|
# 2. THE RESULT WAS ALWAYS {"ok":true}. This branch waited up to 30s for SOCKS
|
||||||
|
# and then ignored the outcome; the `restart` branch slept 3s and claimed
|
||||||
|
# success. So the UI reported "restarted" over a dead daemon.
|
||||||
|
# 3. The underlying torrc was unbindable, so every restart failed anyway —
|
||||||
|
# fixed separately in the torrc generator.
|
||||||
|
#
|
||||||
|
# tor@default is targeted explicitly because it is the unit that carries the
|
||||||
|
# state worth resetting; single-instance installs have only `tor`, hence the
|
||||||
|
# detection and fallback.
|
||||||
|
#
|
||||||
|
# Returns 0 only when SOCKS answers. Callers must not report success without it.
|
||||||
|
restart_tor_daemon() {
|
||||||
|
local unit=tor
|
||||||
|
if systemctl list-unit-files 'tor@*.service' 2>/dev/null | grep -q 'tor@'; then
|
||||||
|
unit=tor@default
|
||||||
|
fi
|
||||||
|
systemctl reset-failed "$unit" 2>/dev/null || true
|
||||||
|
if ! systemctl restart "$unit" 2>/dev/null; then
|
||||||
|
if [ "$unit" != tor ]; then
|
||||||
|
log "restart of $unit failed — falling back to tor.service"
|
||||||
|
systemctl reset-failed tor 2>/dev/null || true
|
||||||
|
systemctl restart tor 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
log "Restarted $unit"
|
||||||
|
|
||||||
|
local i
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
if timeout 1 bash -c 'echo > /dev/tcp/127.0.0.1/9050' 2>/dev/null; then
|
||||||
|
log "Tor SOCKS answering after ${i}s"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
log "ERROR: Tor SOCKS not answering 30s after restarting $unit"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
case "$ACTION_TYPE" in
|
case "$ACTION_TYPE" in
|
||||||
write-torrc-and-restart)
|
write-torrc-and-restart)
|
||||||
if [ ! -f "$TORRC_STAGED" ]; then
|
if [ ! -f "$TORRC_STAGED" ]; then
|
||||||
@@ -64,27 +112,27 @@ case "$ACTION_TYPE" in
|
|||||||
chown debian-tor:debian-tor /etc/tor/torrc 2>/dev/null || true
|
chown debian-tor:debian-tor /etc/tor/torrc 2>/dev/null || true
|
||||||
log "torrc updated from staged file"
|
log "torrc updated from staged file"
|
||||||
|
|
||||||
systemctl restart tor
|
if restart_tor_daemon; then
|
||||||
log "Tor restarted"
|
|
||||||
|
|
||||||
# Wait for SOCKS port
|
|
||||||
for i in $(seq 1 30); do
|
|
||||||
if timeout 1 bash -c 'echo > /dev/tcp/127.0.0.1/9050' 2>/dev/null; then
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
|
|
||||||
sync_hostnames
|
sync_hostnames
|
||||||
write_result '{"ok":true}'
|
write_result '{"ok":true}'
|
||||||
|
else
|
||||||
|
# Never claim success on a dead daemon: the caller surfaces this straight
|
||||||
|
# to the operator, and a false "restarted" is how an outage stays hidden.
|
||||||
|
sync_hostnames
|
||||||
|
write_result '{"ok":false,"error":"Tor restarted but SOCKS never came up on 127.0.0.1:9050 — check journalctl -u tor@default"}'
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
|
|
||||||
restart)
|
restart)
|
||||||
systemctl restart tor
|
if restart_tor_daemon; then
|
||||||
log "Tor restarted"
|
|
||||||
sleep 3
|
|
||||||
sync_hostnames
|
sync_hostnames
|
||||||
write_result '{"ok":true}'
|
write_result '{"ok":true}'
|
||||||
|
else
|
||||||
|
sync_hostnames
|
||||||
|
write_result '{"ok":false,"error":"Tor restarted but SOCKS never came up on 127.0.0.1:9050 — check journalctl -u tor@default"}'
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
|
|
||||||
delete-service)
|
delete-service)
|
||||||
|
|||||||
Reference in New Issue
Block a user