archy/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
01-federation-mesh-hardening	20	execute	1		scripts/container-doctor.sh	false	FED-09	truths prohibitions The container doctor no longer restarts Tor on every run — a node left alone shows Tor uptime growing past the doctor's 5-minute timer interval A hidden-service directory at mode 2700 (Tor's own setting) is recognised as correct and triggers no chmod and no restart A genuinely insecure hidden-service directory (group- or other-readable, e.g. 750 or 707) is still corrected Even when a real permission fix IS applied, Tor cannot be restarted more than once per backoff window, so no future defect can reproduce a restart storm Tor retains its consensus/HSDir cache long enough to resolve .onion addresses, so the mesh Tor fallback works MUST NOT loosen hidden-service directory permissions — group and other access must remain denied MUST NOT disable the doctor's other fixes or the timer itself

Mesh sends fail entirely on affected nodes because both transports are down, and the second failure is self-inflicted.

Diagnosed 2026-07-31 on a live node:

1. The FIPS direct transport (Yggdrasil-style fd.. IPv6) times out with connect_fail for peers other than the currently-connected tree peers, so every send falls back to Tor.
2. The Tor fallback then fails with No more HSDir available to query — Tor cannot resolve any peer .onion address.

Root cause of (2): the container doctor restarts Tor every ~5 minutes, forever. fix_tor_permissions() in scripts/container-doctor.sh treats any mode other than the literal string 700 as broken:

perms=$(stat -c '%a' "$dir")
if [ "$perms" != "700" ]; then
    chmod 700 "$dir"; fixed=true
fi
...
if $fixed; then systemctl restart tor@default; fi

Tor sets its own HiddenServiceDir to 2700 (setgid). So every run the doctor sees 2700 != 700, "fixes" it, and restarts Tor; Tor comes back up and sets 2700 again; the timer fires 5 minutes later and the cycle repeats. Observed restarts on the node: 13:07:56 → 13:13:14 → 13:18:39 → 13:23:57, each within a second of an archipelago-doctor.timer firing. Tor never survives long enough to build a usable consensus/HSDir cache, so onion lookups fail and the mesh's only remaining transport dies with it.

2700 is not a defect — the setgid bit is harmless here and group/other access is still fully denied, which is the property that actually matters.

Scope note: this plan fixes the restart loop only. The FIPS direct-transport connect_fail (problem 1) is a separate concern and belongs with FED-03's structured review of the transport/dial layer — record it there, do not attempt both here.

<execution_context> @$HOME/.claude/gsd-core/workflows/execute-plan.md @$HOME/.claude/gsd-core/templates/summary.md </execution_context>

Task 1: Stop the doctor from fighting Tor over the setgid bit Contained to one shell function; revert is a single-file change. scripts/container-doctor.sh - `scripts/container-doctor.sh` — `fix_tor_permissions()` (~lines 139-164) and how other `fix_*` functions signal "changed" vs "no drift" - `image-recipe/configs/archipelago-doctor.timer` — `OnUnitActiveSec=5min`, `RandomizedDelaySec=60`: this is the loop's clock - `core/archipelago/src/bootstrap.rs:24-31` — the script is embedded via `include_str!` and written to `/home/archipelago/archy/scripts/container-doctor.sh` on every boot, so nodes pick the fix up through the normal binary release Correct the permission predicate so it tests the property that matters — group and other have no access — instead of exact-matching one octal string. Compare the low three digits of `stat -c '%a'` (which omits leading zeros, so handle both `700` and `2700` forms), and treat the directory as correct when those are `700`. Only a genuinely permissive mode (any group or other bit set, e.g. `750`, `707`, `2755`) is a real defect worth fixing.

Keep correcting real defects, and keep restarting Tor when a real fix is
applied — but add a **restart backoff** so a restart storm is impossible even
if some future condition makes the fix fire repeatedly: record the last
restart time (e.g. a timestamp file under `/var/lib/archipelago/`) and skip
the restart if one happened within the last 30 minutes, logging that it was
skipped. The current defect is being fixed at the predicate, but the backoff
is what makes the class of failure non-recurring.

Log clearly in both directions — when a directory is accepted as already
correct (at debug level, so a healthy node stays quiet) and when a real fix
is applied. The original bug was invisible precisely because "Fixed
permissions on ... (2700 -> 700)" looked like the doctor working correctly.

bash -n scripts/container-doctor.sh && sudo bash -c 'set -e; d=$(mktemp -d); mkdir -p "$d/hidden_service_test"; chmod 2700 "$d/hidden_service_test"; stat -c "%a" "$d/hidden_service_test"' - `bash -n scripts/container-doctor.sh` passes - A directory at mode `2700` is accepted: no chmod, no restart, `fixed` stays false - A directory at mode `750` or `707` is still corrected to deny group/other - A second real fix within the backoff window logs a skip instead of restarting Tor - The doctor's other fixes and the timer are untouched The doctor recognises Tor's own 2700 as correct, so it stops restarting Tor every five minutes, and a backoff prevents any future restart storm. Task 2: Confirm Tor stays up and mesh sends recover The doctor no longer mistakes Tor's setgid `2700` hidden-service directory for a permission defect, so it stops chmod-ing it and restarting Tor every ~5 minutes. A restart backoff makes a restart storm impossible even if some other condition triggers the fix repeatedly. This ships to affected nodes via the **normal OTA release only**. Never deploy directly to a user's device (`archy-x250-mad2` or any node that is not ours). On an affected node, after the release lands: 1. `systemctl status tor@default` — note the uptime. Wait 15 minutes (three doctor intervals) and check again: uptime should keep growing, with no restart. Before the fix it reset roughly every 5 minutes. 2. `journalctl -u archipelago-doctor -n 50` — no recurring "Fixed permissions on ... hidden_service_* (2700 -> 700)" lines. 3. Once Tor has been up ~20-30 minutes, confirm onion resolution works: a mesh send to a peer reachable only via Tor should succeed, and the logs should no longer show `No more HSDir available to query`. 4. Confirm the doctor still does its job: temporarily `chmod 750` a hidden-service directory, wait for the next doctor run, and check it is corrected back to deny group/other access. Type "approved", or describe what you saw — which step, what happened instead. - `bash -n` passes; the 2700 case is accepted and the 750/707 cases are still fixed - On an affected node, Tor uptime exceeds the doctor's interval and onion resolution recovers

<success_criteria>

• Tor is no longer restarted every ~5 minutes by the doctor
• Tor keeps its consensus/HSDir cache, so .onion peers resolve and the mesh Tor fallback works again
• Genuinely insecure hidden-service directory permissions are still corrected
• A restart backoff makes this class of failure non-recurring </success_criteria>

Create `.planning/phases/01-federation-mesh-hardening/01-20-SUMMARY.md`. It MUST record the corrected predicate, the backoff mechanism and window, and a note handing the FIPS direct-transport `connect_fail` (problem 1 of the original diagnosis) to FED-03's transport/dial-layer review.