docs(tracker): record the failed-unit self-healing gap observed live on .228
fedimint sat 'failed' for 7+ hours (exit 255 at 21:21, pre-gate) — the reconciler repairs missing/drifted containers but never reset-fails + starts a failed quadlet .service, and the health monitor can't see an app whose container is gone. Concrete lever added to the Tier 2 container-flapping item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
11a4f2910a
commit
804874a78b
@ -128,6 +128,13 @@ those are marked ✅ below with the commit that did it, so we stop re-litigating
|
||||
- **Reconciler churn observability** — no metric/log today distinguishes "settling
|
||||
after restart" from "flapping"; add a per-app restart counter + log line when an
|
||||
app restarts >N times in M minutes so thrash is visible instead of anecdotal.
|
||||
- **Failed-unit self-healing gap (observed live 2026-07-06 on .228)**: fedimint's
|
||||
quadlet unit exited 255 at 21:21 and sat `failed` for 7+ hours — the reconciler
|
||||
never revived it (it repairs missing/drifted containers but doesn't
|
||||
`reset-failed`+start failed .services). Same for the indeedhub trio after the
|
||||
gate run. The health monitor also can't help (container is gone when the unit
|
||||
fails). Add a reconcile step: quadlet-backed app whose .service is `failed` and
|
||||
not user-stopped → reset-failed + start, with backoff.
|
||||
- Already landed, don't re-do: boot-reconciler circuit breaker (2026-07-01),
|
||||
indeedhub crashloop fix (2026-07-01), async blocking-Command pass (`4c75bb3d`,
|
||||
removes executor stalls that made the API janky under reconcile load).
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user