archy/.planning/phases/01-federation-mesh-hardening/01-20-PLAN.md

---
phase: 01-federation-mesh-hardening
plan: 20
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/container-doctor.sh
autonomous: false
requirements: [FED-09]

must_haves:
  truths:
    - "The container doctor no longer restarts Tor on every run — a node left alone shows Tor uptime growing past the doctor's 5-minute timer interval"
    - "A hidden-service directory at mode 2700 (Tor's own setting) is recognised as correct and triggers no chmod and no restart"
    - "A genuinely insecure hidden-service directory (group- or other-readable, e.g. 750 or 707) is still corrected"
    - "Even when a real permission fix IS applied, Tor cannot be restarted more than once per backoff window, so no future defect can reproduce a restart storm"
    - "Tor retains its consensus/HSDir cache long enough to resolve .onion addresses, so the mesh Tor fallback works"
  prohibitions:
    - "MUST NOT loosen hidden-service directory permissions — group and other access must remain denied"
    - "MUST NOT disable the doctor's other fixes or the timer itself"
---

<objective>
Mesh sends fail entirely on affected nodes because both transports are down, and
the second failure is self-inflicted.

Diagnosed 2026-07-31 on a live node:
1. The FIPS direct transport (Yggdrasil-style `fd..` IPv6) times out with
   `connect_fail` for peers other than the currently-connected tree peers, so
   every send falls back to Tor.
2. The Tor fallback then fails with `No more HSDir available to query` — Tor
   cannot resolve any peer `.onion` address.

Root cause of (2): **the container doctor restarts Tor every ~5 minutes,
forever.** `fix_tor_permissions()` in `scripts/container-doctor.sh` treats any
mode other than the literal string `700` as broken:

```sh
perms=$(stat -c '%a' "$dir")
if [ "$perms" != "700" ]; then
    chmod 700 "$dir"; fixed=true
fi
...
if $fixed; then systemctl restart tor@default; fi
```

Tor sets its own `HiddenServiceDir` to **2700** (setgid). So every run the doctor
sees `2700 != 700`, "fixes" it, and restarts Tor; Tor comes back up and sets 2700
again; the timer fires 5 minutes later and the cycle repeats. Observed restarts
on the node: 13:07:56 → 13:13:14 → 13:18:39 → 13:23:57, each within a second of
an `archipelago-doctor.timer` firing. Tor never survives long enough to build a
usable consensus/HSDir cache, so onion lookups fail and the mesh's only remaining
transport dies with it.

`2700` is not a defect — the setgid bit is harmless here and group/other access
is still fully denied, which is the property that actually matters.

**Scope note:** this plan fixes the restart loop only. The FIPS direct-transport
`connect_fail` (problem 1) is a separate concern and belongs with FED-03's
structured review of the transport/dial layer — record it there, do not attempt
both here.
</objective>

<execution_context>
@$HOME/.claude/gsd-core/workflows/execute-plan.md
@$HOME/.claude/gsd-core/templates/summary.md
</execution_context>

<tasks>

<task type="auto">
  <name>Task 1: Stop the doctor from fighting Tor over the setgid bit</name>
  <reversibility rating="reversible">Contained to one shell function; revert is a single-file change.</reversibility>
  <files>scripts/container-doctor.sh</files>
  <read_first>
    - `scripts/container-doctor.sh` — `fix_tor_permissions()` (~lines 139-164) and how other `fix_*` functions signal "changed" vs "no drift"
    - `image-recipe/configs/archipelago-doctor.timer` — `OnUnitActiveSec=5min`, `RandomizedDelaySec=60`: this is the loop's clock
    - `core/archipelago/src/bootstrap.rs:24-31` — the script is embedded via `include_str!` and written to `/home/archipelago/archy/scripts/container-doctor.sh` on every boot, so nodes pick the fix up through the normal binary release
  </read_first>
  <action>
    Correct the permission predicate so it tests the property that matters —
    group and other have no access — instead of exact-matching one octal string.
    Compare the low three digits of `stat -c '%a'` (which omits leading zeros, so
    handle both `700` and `2700` forms), and treat the directory as correct when
    those are `700`. Only a genuinely permissive mode (any group or other bit
    set, e.g. `750`, `707`, `2755`) is a real defect worth fixing.

    Keep correcting real defects, and keep restarting Tor when a real fix is
    applied — but add a **restart backoff** so a restart storm is impossible even
    if some future condition makes the fix fire repeatedly: record the last
    restart time (e.g. a timestamp file under `/var/lib/archipelago/`) and skip
    the restart if one happened within the last 30 minutes, logging that it was
    skipped. The current defect is being fixed at the predicate, but the backoff
    is what makes the class of failure non-recurring.

    Log clearly in both directions — when a directory is accepted as already
    correct (at debug level, so a healthy node stays quiet) and when a real fix
    is applied. The original bug was invisible precisely because "Fixed
    permissions on ... (2700 -> 700)" looked like the doctor working correctly.
  </action>
  <verify>
    <automated>bash -n scripts/container-doctor.sh && sudo bash -c 'set -e; d=$(mktemp -d); mkdir -p "$d/hidden_service_test"; chmod 2700 "$d/hidden_service_test"; stat -c "%a" "$d/hidden_service_test"' </automated>
  </verify>
  <acceptance_criteria>
    - `bash -n scripts/container-doctor.sh` passes
    - A directory at mode `2700` is accepted: no chmod, no restart, `fixed` stays false
    - A directory at mode `750` or `707` is still corrected to deny group/other
    - A second real fix within the backoff window logs a skip instead of restarting Tor
    - The doctor's other fixes and the timer are untouched
  </acceptance_criteria>
  <done>The doctor recognises Tor's own 2700 as correct, so it stops restarting Tor every five minutes, and a backoff prevents any future restart storm.</done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 2: Confirm Tor stays up and mesh sends recover</name>
  <what-built>
    The doctor no longer mistakes Tor's setgid `2700` hidden-service directory for
    a permission defect, so it stops chmod-ing it and restarting Tor every ~5
    minutes. A restart backoff makes a restart storm impossible even if some other
    condition triggers the fix repeatedly.
  </what-built>
  <constraint priority="highest">
    This ships to affected nodes via the **normal OTA release only**. Never deploy
    directly to a user's device (`archy-x250-mad2` or any node that is not ours).
  </constraint>
  <how-to-verify>
    On an affected node, after the release lands:
    1. `systemctl status tor@default` — note the uptime. Wait 15 minutes (three
       doctor intervals) and check again: uptime should keep growing, with no
       restart. Before the fix it reset roughly every 5 minutes.
    2. `journalctl -u archipelago-doctor -n 50` — no recurring
       "Fixed permissions on ... hidden_service_* (2700 -> 700)" lines.
    3. Once Tor has been up ~20-30 minutes, confirm onion resolution works: a
       mesh send to a peer reachable only via Tor should succeed, and the logs
       should no longer show `No more HSDir available to query`.
    4. Confirm the doctor still does its job: temporarily `chmod 750` a
       hidden-service directory, wait for the next doctor run, and check it is
       corrected back to deny group/other access.
  </how-to-verify>
  <resume-signal>Type "approved", or describe what you saw — which step, what happened instead.</resume-signal>
</task>

</tasks>

<verification>
- `bash -n` passes; the 2700 case is accepted and the 750/707 cases are still fixed
- On an affected node, Tor uptime exceeds the doctor's interval and onion resolution recovers
</verification>

<success_criteria>
- Tor is no longer restarted every ~5 minutes by the doctor
- Tor keeps its consensus/HSDir cache, so `.onion` peers resolve and the mesh Tor fallback works again
- Genuinely insecure hidden-service directory permissions are still corrected
- A restart backoff makes this class of failure non-recurring
</success_criteria>

<output>
Create `.planning/phases/01-federation-mesh-hardening/01-20-SUMMARY.md`. It MUST record the corrected predicate, the backoff mechanism and window, and a note handing the FIPS direct-transport `connect_fail` (problem 1 of the original diagnosis) to FED-03's transport/dial-layer review.
</output>