lfg2025/archy
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases 44 Wiki Activity

docs(tracker): record the failed-unit self-healing gap observed live on .228

Browse Source 
fedimint sat 'failed' for 7+ hours (exit 255 at 21:21, pre-gate) — the
reconciler repairs missing/drifted containers but never reset-fails +
starts a failed quadlet .service, and the health monitor can't see an
app whose container is gone. Concrete lever added to the Tier 2
container-flapping item.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago 2026-07-06 01:44:21 -04:00
parent 11a4f2910a
commit 804874a78b
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/UNIFIED-TASK-TRACKER.md
View File

@ -128,6 +128,13 @@ those are marked ✅ below with the commit that did it, so we stop re-litigating
- **Reconciler churn observability** — no metric/log today distinguishes "settling - **Reconciler churn observability** — no metric/log today distinguishes "settling
after restart" from "flapping"; add a per-app restart counter + log line when an after restart" from "flapping"; add a per-app restart counter + log line when an
app restarts >N times in M minutes so thrash is visible instead of anecdotal. app restarts >N times in M minutes so thrash is visible instead of anecdotal.
- **Failed-unit self-healing gap (observed live 2026-07-06 on .228)**: fedimint's
quadlet unit exited 255 at 21:21 and sat `failed` for 7+ hours — the reconciler
never revived it (it repairs missing/drifted containers but doesn't
`reset-failed`+start failed .services). Same for the indeedhub trio after the
gate run. The health monitor also can't help (container is gone when the unit
fails). Add a reconcile step: quadlet-backed app whose .service is `failed` and
not user-stopped → reset-failed + start, with backoff.
- Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01), - Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01),
indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`, indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`,
removes executor stalls that made the API janky under reconcile load). removes executor stalls that made the API janky under reconcile load).