lfg2025/archy
1
0
Fork 0
Code Issues 25 Pull Requests 4 Actions Packages Projects Releases 74 Wiki Activity

refactor(container): move companion UIs to systemd via Quadlet

Browse Source 
Companion UI containers (archy-bitcoin-ui, archy-lnd-ui,
archy-electrs-ui) used to be launched as fire-and-forget tokio::spawn
blocks from install.rs. If archipelago crashed mid-spawn or the
container's cgroup was reaped, companions vanished from podman ps -a
and only a manual rm/run could bring them back (the .228 incident).

Now each companion is rendered as a Quadlet .container unit under
~/.config/containers/systemd/, daemon-reloaded, and started via
systemctl --user. systemd owns supervision from that point on:

- archipelago can crash, restart, or be uninstalled without touching
  any companion.
- Quadlet's Restart=always + RestartSec=10 handles container exits.
- A 30s reconcile tick in boot_reconciler enumerates expected
  companion units and re-installs any whose unit file or service
  vanished — defense-in-depth against external tampering.

New module layout:
- container/quadlet.rs: pure unit renderer + atomic write_if_changed
  + systemctl helpers (daemon_reload_user / enable_now / disable_remove
  / is_active). 6 unit tests, no I/O in the renderer.
- container/companion.rs: per-app companion specs, install/remove/
  reconcile, image presence (build local first, fall back to insecure
  registry only via image_uses_insecure_registry whitelist). 2 tests.

install.rs handle_package_install now ends with a single call to
companion::install_for(package_id), replacing 287 lines of spawn-and-
hope shellouts plus a ~120-line nginx auth-injector helper that worked
around per-node RPC password baking. The helper is gone too — the
pre-start hook renders the per-node nginx.conf to /var/lib/archipelago/
bitcoin-ui/nginx.conf and the Quadlet unit bind-mounts it read-only.

runtime.rs handle_package_uninstall now disables companions before
the container rm loop. Otherwise systemd's Restart=always would
respawn each companion within ~10s of removal.

Tests: 53 container tests pass, including 6 quadlet renderer tests
(host network, bridge network, capability set, atomic write idempotence)
and 2 companion specs (per-app companion lookup, build_unit shape).
boot_reconciler tests gain a #[cfg(test)] without_companion_stage()
flag so the paused-clock fixtures don't race the real systemctl I/O.

A bats regression test (companion-survives-archipelago-restart.bats,
gated on ARCHY_ALLOW_DESTRUCTIVE=1) asserts the .228 failure mode
cannot recur: every installed companion has a unit file, services
stay active across systemctl --user restart archipelago, and a
deleted unit file is recreated within one reconcile tick.

Net delta: +941 / -363, but the +941 is mostly tests (~440 lines)
and the new declarative layer; the imperative tokio::spawn block and
its nginx-auth helper are gone, removing two failure classes
(orphan companions on archipelago crash, and post-start exec races
under tightly-confined cgroups) that previously needed manual SSH
recovery.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-05-01 10:45:07 -04:00
co-authored by Claude Opus 4.7
parent 2bf8181110
commit 23c4e7441f
9 changed files with 941 additions and 363 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/archipelago/src/container/boot_reconciler.rs
+52 -10
View File
@@ -29,6 +29,11 @@ pub struct BootReconciler {
orchestrator: Arc<ProdContainerOrchestrator>,
interval: Duration,
shutdown: Arc<Notify>,
/// Run the companion-unit repair stage each tick. Default true.
/// Tests disable this — companion reconcile shells out to
/// `systemctl --user` and `podman`, which both block real time
/// and would race the paused-clock test fixtures.
companion_stage: bool,
}
impl BootReconciler {
@@ -41,28 +46,46 @@ impl BootReconciler {
orchestrator,
interval,
shutdown,
companion_stage: true,
}
}
/// Disable the companion-unit reconcile stage. Used by unit tests
/// that exercise loop cadence without the real systemd / podman
/// surface. Production must not call this.
#[cfg(test)]
pub fn without_companion_stage(mut self) -> Self {
self.companion_stage = false;
self
}
/// Run the reconcile loop until `shutdown` is notified.
///
/// Does one reconcile immediately, then sleeps `interval` between
/// subsequent passes. A `shutdown.notify_one()` call unblocks the sleep
/// and the task returns after the *next* pass completes.
///
/// Never panics: per-app failures are absorbed into `ReconcileReport` by
/// the orchestrator, and `reconcile_all` itself returns infallibly.
/// Each pass is two stages:
/// 1. App reconcile: `reconcile_all()` keeps every loaded manifest's
/// container running.
/// 2. Companion reconcile: any expected Quadlet companion unit that
/// is missing or inactive is repaired (writes the unit, daemon-
/// reloads, starts the service). This is the safety net for the
/// "someone deleted my unit file" / "systemd lost the service"
/// failure modes.
///
/// Never panics: per-app failures are absorbed into `ReconcileReport`
/// by the orchestrator, and companion failures are logged but never
/// propagated.
pub async fn run_forever(self) {
// Initial pass: no delay.
let report = self.orchestrator.reconcile_all().await;
Self::log_report(&report);
self.tick().await;
loop {
let deadline = Instant::now() + self.interval;
tokio::select! {
_ = time::sleep_until(deadline) => {
let report = self.orchestrator.reconcile_all().await;
Self::log_report(&report);
self.tick().await;
}
_ = self.shutdown.notified() => {
tracing::info!("boot reconciler: shutdown requested, exiting loop");
@@ -72,6 +95,25 @@ impl BootReconciler {
}
}
async fn tick(&self) {
let report = self.orchestrator.reconcile_all().await;
Self::log_report(&report);
if !self.companion_stage {
return;
}
let installed = self.orchestrator.manifest_ids().await;
for (companion, err) in
crate::container::companion::reconcile(&installed).await
{
tracing::warn!(
companion = %companion,
error = %err,
"companion reconcile failed"
);
}
}
fn log_report(report: &ReconcileReport) {
for (app_id, action) in &report.actions {
tracing::debug!(app_id = %app_id, action = ?action, "reconcile action");
@@ -218,7 +260,7 @@ mod tests {
let orch = orch_with_one_running_manifest(rt.clone()).await;
let shutdown = Arc::new(Notify::new());
let reconciler =
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone());
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone()).without_companion_stage();
let handle = tokio::spawn(reconciler.run_forever());
// Yield so the spawned task gets CPU to run its initial reconcile.
@@ -242,7 +284,7 @@ mod tests {
let orch = orch_with_one_running_manifest(rt.clone()).await;
let shutdown = Arc::new(Notify::new());
let reconciler =
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone());
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone()).without_companion_stage();
let handle = tokio::spawn(reconciler.run_forever());
tokio::task::yield_now().await;
@@ -271,7 +313,7 @@ mod tests {
let orch = orch_with_one_running_manifest(rt.clone()).await;
let shutdown = Arc::new(Notify::new());
let reconciler =
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone());
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone()).without_companion_stage();
let handle = tokio::spawn(reconciler.run_forever());
tokio::task::yield_now().await;
tokio::task::yield_now().await;
@@ -305,7 +347,7 @@ mod tests {
.await;
let shutdown = Arc::new(Notify::new());
let reconciler =
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone());
BootReconciler::new(orch.clone(), Duration::from_secs(30), shutdown.clone()).without_companion_stage();
let handle = tokio::spawn(reconciler.run_forever());
tokio::task::yield_now().await;