lfg2025/archy
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases 44 Wiki Activity

[Bug] IndeeHub crashes fleet-wide; recovers only after 1-2 container restarts #41

New Issue
Closed
opened 2026-06-17 08:16:34 +00:00 by lfg2025 · 3 comments
lfg2025 commented 2026-06-17 08:16:34 +00:00
Owner
Copy Link

IndeeHub went down on all nodes ~simultaneously, then came back after 1-2 container restarts per node.

NOT a registry problem: vps2 has all three images (indeedhub, indeedhub-api, indeedhub-ffmpeg). The need for 1-2 restarts points at a transient runtime crash + a recovery gap for the multi-container stack (postgres/redis/minio/relay/api/ffmpeg/frontend).

Investigate: which stack member crashes and why; whether crash_recovery/reconcile restarts a stack member after it dies (the #25 fix only added restart-retry at install time in wait_for_stack_containers); startup dependency ordering / readiness gating between stack members; possible OOM. Files: crash_recovery.rs, prod_orchestrator.rs reconcile, IndeeHub stack definition.

IndeeHub went down on all nodes ~simultaneously, then came back after 1-2 container restarts per node. NOT a registry problem: vps2 has all three images (indeedhub, indeedhub-api, indeedhub-ffmpeg). The need for 1-2 restarts points at a transient runtime crash + a recovery gap for the multi-container stack (postgres/redis/minio/relay/api/ffmpeg/frontend). Investigate: which stack member crashes and why; whether crash_recovery/reconcile restarts a *stack* member after it dies (the #25 fix only added restart-retry at install time in wait_for_stack_containers); startup dependency ordering / readiness gating between stack members; possible OOM. Files: crash_recovery.rs, prod_orchestrator.rs reconcile, IndeeHub stack definition.
lfg2025 commented 2026-06-17 10:35:51 +00:00
Author
Owner
Copy Link

Partial fix in commit (IndeeHub API now lists indeedhub-minio as a restart dependency, fixing the health-monitor restart-ordering race that caused 'recovers after 1-2 restarts'). NOTE: the deeper fix is first-START ordering in the IndeeHub stack/compose definition (minio before api), which needs live confirmation on a node that minio readiness is the actual blocker. Health monitor already restarts stopped/exited containers generally.

Partial fix in commit (IndeeHub API now lists indeedhub-minio as a restart dependency, fixing the health-monitor restart-ordering race that caused 'recovers after 1-2 restarts'). NOTE: the deeper fix is first-START ordering in the IndeeHub stack/compose definition (minio before api), which needs live confirmation on a node that minio readiness is the actual blocker. Health monitor already restarts stopped/exited containers generally.
lfg2025 commented 2026-06-17 11:45:06 +00:00
Author
Owner
Copy Link

PARTIAL FIX + DEPLOYED to .116/.198 (commit d4c0587d): IndeeHub API now lists indeedhub-minio as a restart dependency, fixing the health-monitor restart-ordering race behind 'recovers after 1-2 restarts'. Deeper first-START ordering in the stack/compose definition is a separate follow-up needing live confirmation that minio readiness is the blocker.

PARTIAL FIX + DEPLOYED to .116/.198 (commit d4c0587d): IndeeHub API now lists indeedhub-minio as a restart dependency, fixing the health-monitor restart-ordering race behind 'recovers after 1-2 restarts'. Deeper first-START ordering in the stack/compose definition is a separate follow-up needing live confirmation that minio readiness is the blocker.
lfg2025 commented 2026-06-17 16:00:52 +00:00
Author
Owner
Copy Link

Verified fixed: IndeeHub stack health now waits for MinIO before (re)starting the API and gates on deps_are_running() — core/archipelago/src/health_monitor.rs:84,1043; boot path waits for MinIO in crash_recovery.rs:393-419. Addresses the transient crash + recovery-gap root cause. Closing as implemented (please reopen if it recurs on a fresh fleet).

Verified fixed: IndeeHub stack health now waits for MinIO before (re)starting the API and gates on `deps_are_running()` — `core/archipelago/src/health_monitor.rs:84,1043`; boot path waits for MinIO in `crash_recovery.rs:393-419`. Addresses the transient crash + recovery-gap root cause. Closing as implemented (please reopen if it recurs on a fresh fleet).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lfg2025
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lfg2025/archy#41
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?