lfg2025/archy
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases 44 Wiki Activity
archy/core/archipelago/src/container/mod.rs

20 lines
558 B
Rust
Raw Normal View History

feat(container): bitcoin-ui pre-start hook renders nginx.conf from embedded template Replaces the first-boot-containers.sh sed/envsubst approach with a Rust-native render step bound into the ContainerOrchestrator lifecycle. - New container::bitcoin_ui module: embeds the nginx.conf template via include_str!, reads the plaintext RPC password from /var/lib/archipelago/secrets/bitcoin-rpc-password, substitutes {{BITCOIN_RPC_AUTH}} with base64(archipelago:<password>), and atomic- writes (tmp + rename) to /var/lib/archipelago/bitcoin-ui/nginx.conf. Idempotent: byte-compares before writing so unchanged input is a no-op (no inode churn, no restart cascade). - ProdContainerOrchestrator gains run_pre_start_hooks(app_id) returning HookOutcome::{Rewritten, Unchanged}. Fires in install_fresh before create_container, and in ensure_running: on Running + Rewritten triggers a restart; on Stopped re-renders then starts. - bitcoin-ui Dockerfile no longer COPYs a default.conf; the file now arrives via runtime bind-mount of the rendered config. If the bind- mount is ever missing, nginx starts with no site configured and returns 404 everywhere — safe failure vs. serving upstream RPC with a stale Authorization header. - apps/{bitcoin,electrs,lnd}-ui/manifest.yml land as first-class manifests. bitcoin-ui declares the bind-mount target and a dependency on bitcoin-core; electrs-ui and lnd-ui declare their own deps and health checks. - 8 new unit tests on the render fn (idempotency, rotation, trimming, missing/empty secret, template invariants) plus an integration test asserting install(bitcoin-ui) actually lands a substituted nginx.conf on disk via the hook. 39/39 container:: tests pass (test_parse_image_versions pre-existing failure unchanged, out of scope).
2026-04-23 02:19:52 -04:00
pub mod bitcoin_ui;
feat(container): BootReconciler — periodic reconcile loop for prod orchestrator Step 5 of the rust-orchestrator migration. New file boot_reconciler.rs holds a small Tokio task that calls ProdContainerOrchestrator::reconcile_all() on a 30-second cadence (answered design Q3). * BootReconciler::new(orch, interval, shutdown) — shutdown is an Arc<Notify> so callers can trigger a graceful exit without pulling in tokio-util. * run_forever(self) — does one reconcile immediately, then loops on tokio::select! { sleep_until | shutdown.notified() }. Shutdown interrupts the sleep but never an in-flight reconcile_all call. * Per-pass outcomes are logged at debug/warn; failures never propagate out because reconcile_all already absorbs per-app errors into ReconcileReport. Four tokio::test(start_paused = true) tests verify the loop cadence against a CountingRuntime test double: * initial_pass_fires_immediately — first reconcile runs with no delay * second_pass_fires_after_interval — second pass fires after exactly interval elapses in paused-clock time * shutdown_terminates_loop — notify_one() lets run_forever return * failure_in_one_pass_does_not_stop_loop — the loop keeps ticking even when the first pass had to install a missing container Not wired into main.rs yet — that is Step 6. Re-exported from container::mod as BootReconciler + RECONCILER_DEFAULT_INTERVAL for the wire-up step.
2026-04-22 19:04:34 -04:00
pub mod boot_reconciler;
refactor(container): move companion UIs to systemd via Quadlet Companion UI containers (archy-bitcoin-ui, archy-lnd-ui, archy-electrs-ui) used to be launched as fire-and-forget tokio::spawn blocks from install.rs. If archipelago crashed mid-spawn or the container's cgroup was reaped, companions vanished from podman ps -a and only a manual rm/run could bring them back (the .228 incident). Now each companion is rendered as a Quadlet .container unit under ~/.config/containers/systemd/, daemon-reloaded, and started via systemctl --user. systemd owns supervision from that point on: - archipelago can crash, restart, or be uninstalled without touching any companion. - Quadlet's Restart=always + RestartSec=10 handles container exits. - A 30s reconcile tick in boot_reconciler enumerates expected companion units and re-installs any whose unit file or service vanished — defense-in-depth against external tampering. New module layout: - container/quadlet.rs: pure unit renderer + atomic write_if_changed + systemctl helpers (daemon_reload_user / enable_now / disable_remove / is_active). 6 unit tests, no I/O in the renderer. - container/companion.rs: per-app companion specs, install/remove/ reconcile, image presence (build local first, fall back to insecure registry only via image_uses_insecure_registry whitelist). 2 tests. install.rs handle_package_install now ends with a single call to companion::install_for(package_id), replacing 287 lines of spawn-and- hope shellouts plus a ~120-line nginx auth-injector helper that worked around per-node RPC password baking. The helper is gone too — the pre-start hook renders the per-node nginx.conf to /var/lib/archipelago/ bitcoin-ui/nginx.conf and the Quadlet unit bind-mounts it read-only. runtime.rs handle_package_uninstall now disables companions before the container rm loop. Otherwise systemd's Restart=always would respawn each companion within ~10s of removal. Tests: 53 container tests pass, including 6 quadlet renderer tests (host network, bridge network, capability set, atomic write idempotence) and 2 companion specs (per-app companion lookup, build_unit shape). boot_reconciler tests gain a #[cfg(test)] without_companion_stage() flag so the paused-clock fixtures don't race the real systemctl I/O. A bats regression test (companion-survives-archipelago-restart.bats, gated on ARCHY_ALLOW_DESTRUCTIVE=1) asserts the .228 failure mode cannot recur: every installed companion has a unit file, services stay active across systemctl --user restart archipelago, and a deleted unit file is recreated within one reconcile tick. Net delta: +941 / -363, but the +941 is mostly tests (~440 lines) and the new declarative layer; the imperative tokio::spawn block and its nginx-auth helper are gone, removing two failure classes (orphan companions on archipelago crash, and post-start exec races under tightly-confined cgroups) that previously needed manual SSH recovery. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 10:45:07 -04:00
pub mod companion;
mid coding commit
2026-01-24 22:59:20 +00:00
pub mod data_manager;
pub mod dev_orchestrator;
Enhance Docker integration and API for container management - Implemented Docker container scanning and periodic updates in the Server initialization. - Added new RPC endpoints for managing Docker containers, including start, stop, and restart functionalities. - Updated the API to handle package management for Docker-based applications. - Improved environment variable handling for user-specific configurations in Podman and Docker clients. - Enhanced the development startup script to include Docker container management and provide clearer instructions for full stack setup.
2026-01-27 23:21:26 +00:00
pub mod docker_packages;
feat(orchestrator): complete container migration and release hardening
2026-04-28 15:00:58 -04:00
pub mod filebrowser;
fix: netavark GLIBC mismatch in ISO, container adopt, app updates ISO build no longer copies netavark from build host (Debian 13/GLIBC 2.41) which broke container networking on Debian 12 targets. Rootfs already installs netavark from Debian 12 repos — just configure the backend. Install RPC now adopts existing containers (from first-boot) instead of erroring on duplicates. Container scanner extracts real versions from image tags and detects available updates against pinned versions. Frontend shows update button with version info when updates are available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-09 11:47:35 +02:00
pub mod image_versions;
chore(release): stage v1.7.52-alpha
2026-05-05 11:29:18 -04:00
pub mod lnd;
feat(container): ProdContainerOrchestrator with build-or-pull, adoption, reconcile Step 3 of the rust-orchestrator-migration. New file prod_orchestrator.rs (999 LOC) implements the full public surface that will replace scripts/first-boot-containers.sh: * install / start / stop / restart / remove / upgrade / status / list / logs / health * adopt_existing: read-only scan that claims containers matching our manifests by name, without recreating — preserves the v1.7.42 fixture on .116. * reconcile_all: level-triggered, per-app failures collected rather than aborting. * install_fresh: build-or-pull (Step 2 trait methods), relative build contexts resolved against the manifest directory. Naming rule (answered design Q1): UI app IDs (bitcoin-ui/electrs-ui/lnd-ui) get the archy- prefix; backends keep their bare ID. An explicit extensions.container_name always wins. Codified in compute_container_name() with unit tests for all three tiers. Concurrency (answered design Q4): per-app tokio::sync::Mutex<()> created lazily, protecting every mutating op against the reconciler loop. Acquiring the per-app lock only needs a read lock on the map, so independent apps do not serialize. 16 tests: 3 sync naming rule tests + 13 tokio async tests covering install (pull, build-absent, build-present, relative-context), reconcile (noop/exited/missing/ mixed-failure), adopt-by-name, upgrade sequence ordering, list filtering, health state mapping, and unknown-app-id rejection. All pass. Not wired into main.rs yet — that is Step 6. Crate builds clean with expected unused warnings for the new re-exports.
2026-04-22 18:32:31 -04:00
pub mod prod_orchestrator;
refactor(container): move companion UIs to systemd via Quadlet Companion UI containers (archy-bitcoin-ui, archy-lnd-ui, archy-electrs-ui) used to be launched as fire-and-forget tokio::spawn blocks from install.rs. If archipelago crashed mid-spawn or the container's cgroup was reaped, companions vanished from podman ps -a and only a manual rm/run could bring them back (the .228 incident). Now each companion is rendered as a Quadlet .container unit under ~/.config/containers/systemd/, daemon-reloaded, and started via systemctl --user. systemd owns supervision from that point on: - archipelago can crash, restart, or be uninstalled without touching any companion. - Quadlet's Restart=always + RestartSec=10 handles container exits. - A 30s reconcile tick in boot_reconciler enumerates expected companion units and re-installs any whose unit file or service vanished — defense-in-depth against external tampering. New module layout: - container/quadlet.rs: pure unit renderer + atomic write_if_changed + systemctl helpers (daemon_reload_user / enable_now / disable_remove / is_active). 6 unit tests, no I/O in the renderer. - container/companion.rs: per-app companion specs, install/remove/ reconcile, image presence (build local first, fall back to insecure registry only via image_uses_insecure_registry whitelist). 2 tests. install.rs handle_package_install now ends with a single call to companion::install_for(package_id), replacing 287 lines of spawn-and- hope shellouts plus a ~120-line nginx auth-injector helper that worked around per-node RPC password baking. The helper is gone too — the pre-start hook renders the per-node nginx.conf to /var/lib/archipelago/ bitcoin-ui/nginx.conf and the Quadlet unit bind-mounts it read-only. runtime.rs handle_package_uninstall now disables companions before the container rm loop. Otherwise systemd's Restart=always would respawn each companion within ~10s of removal. Tests: 53 container tests pass, including 6 quadlet renderer tests (host network, bridge network, capability set, atomic write idempotence) and 2 companion specs (per-app companion lookup, build_unit shape). boot_reconciler tests gain a #[cfg(test)] without_companion_stage() flag so the paused-clock fixtures don't race the real systemctl I/O. A bats regression test (companion-survives-archipelago-restart.bats, gated on ARCHY_ALLOW_DESTRUCTIVE=1) asserts the .228 failure mode cannot recur: every installed companion has a unit file, services stay active across systemctl --user restart archipelago, and a deleted unit file is recreated within one reconcile tick. Net delta: +941 / -363, but the +941 is mostly tests (~440 lines) and the new declarative layer; the imperative tokio::spawn block and its nginx-auth helper are gone, removing two failure classes (orphan companions on archipelago crash, and post-start exec races under tightly-confined cgroups) that previously needed manual SSH recovery. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 10:45:07 -04:00
pub mod quadlet;
feat: dynamic container registry with fallback Configurable registry list persisted to config/registries.json. Image pulls try all registries in priority order — if primary fails, fallback registries are attempted automatically. RPC endpoints: registry.list, registry.add, registry.remove, registry.test. Replaces hardcoded fallback logic with extensible registry system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 08:09:14 -04:00
pub mod registry;
feat(container): ContainerOrchestrator trait, RpcHandler uses it in prod Step 4 of the rust-orchestrator migration. Unifies the container lifecycle surface behind a single trait so the RPC layer stops caring whether it is talking to the dev or prod orchestrator. * New trait core/archipelago/src/container/traits.rs: ContainerOrchestrator with install / start / stop / restart / remove / upgrade / status / list / logs / health, all keyed by app_id. Every method is async_trait-based. * ProdContainerOrchestrator: the lifecycle methods are moved from inherent impl into the trait impl (avoids name-shadowing recursion). Adoption and reconcile remain inherent since only main.rs / BootReconciler call them. * DevContainerOrchestrator: new trait impl that forwards to the existing Dev-named methods, applying the dev container-name + port-offset rules internally. New load_manifest_for() helper resolves app_id to <data_dir>/apps/<app_id>/manifest.yml so trait-level install(app_id) works in dev too. install_container(manifest, path) stays inherent for the manifest-path RPC shape. * RpcHandler now holds Option<Arc<dyn ContainerOrchestrator>> and, when in dev mode, a separate Option<Arc<DevContainerOrchestrator>> for the manifest_path install RPC. In prod mode RpcHandler::new() constructs a ProdContainerOrchestrator and calls load_manifests() at startup. * All seven container-* RPC guards no longer say dev mode required. container-install still requires dev mode because its manifest_path argument has no prod meaning; every other container RPC now works in both modes via the trait. BOOT STILL DOES NOT USE THIS. main.rs wire-up (Step 6) and BootReconciler (Step 5) come next. Until then the prod orchestrator is constructed but nothing populates /opt/archipelago/apps so it has zero manifests to manage, matching the pre-Step-4 behaviour. Verification: cargo build -p archipelago clean (11 expected unused method warnings for methods not yet wired from main.rs). cargo test -p archipelago: all 21 container::* tests pass (16 prod_orchestrator + 5 others). 24 other test failures are pre-existing and unrelated (identity_manager / session / wallet / mesh / credentials — all independently flaky on file-backed state).
2026-04-22 18:56:52 -04:00
pub mod traits;
mid coding commit
2026-01-24 22:59:20 +00:00
feat(container): BootReconciler — periodic reconcile loop for prod orchestrator Step 5 of the rust-orchestrator migration. New file boot_reconciler.rs holds a small Tokio task that calls ProdContainerOrchestrator::reconcile_all() on a 30-second cadence (answered design Q3). * BootReconciler::new(orch, interval, shutdown) — shutdown is an Arc<Notify> so callers can trigger a graceful exit without pulling in tokio-util. * run_forever(self) — does one reconcile immediately, then loops on tokio::select! { sleep_until | shutdown.notified() }. Shutdown interrupts the sleep but never an in-flight reconcile_all call. * Per-pass outcomes are logged at debug/warn; failures never propagate out because reconcile_all already absorbs per-app errors into ReconcileReport. Four tokio::test(start_paused = true) tests verify the loop cadence against a CountingRuntime test double: * initial_pass_fires_immediately — first reconcile runs with no delay * second_pass_fires_after_interval — second pass fires after exactly interval elapses in paused-clock time * shutdown_terminates_loop — notify_one() lets run_forever return * failure_in_one_pass_does_not_stop_loop — the loop keeps ticking even when the first pass had to install a missing container Not wired into main.rs yet — that is Step 6. Re-exported from container::mod as BootReconciler + RECONCILER_DEFAULT_INTERVAL for the wire-up step.
2026-04-22 19:04:34 -04:00
pub use boot_reconciler::{BootReconciler, DEFAULT_INTERVAL as RECONCILER_DEFAULT_INTERVAL};
mid coding commit
2026-01-24 22:59:20 +00:00
pub use dev_orchestrator::DevContainerOrchestrator;
Enhance Docker integration and API for container management - Implemented Docker container scanning and periodic updates in the Server initialization. - Added new RPC endpoints for managing Docker containers, including start, stop, and restart functionalities. - Updated the API to handle package management for Docker-based applications. - Improved environment variable handling for user-specific configurations in Podman and Docker clients. - Enhanced the development startup script to include Docker container management and provide clearer instructions for full stack setup.
2026-01-27 23:21:26 +00:00
pub use docker_packages::DockerPackageScanner;
refactor: drop dead code surfaced by cargo cargo check was showing five real warnings, all genuinely dead: * container/mod.rs — re-exports compute_container_name, AdoptionReport, ReconcileAction, ReconcileReport were unused outside prod_orchestrator. Drop from the pub use line. * prod_orchestrator — with_runtime + insert_manifest_for_test only exist for the test module in the same file. Mark them #[cfg(test)] so they don't appear in release builds. * async_lifecycle — remove_package_entry has no callers; doc claims "used for install-failure cleanup" but nothing cleans up. Delete (10 lines). * registry.rs — `use tracing::{debug, info};` had no consumers. * fips.rs — unused-assignment chain on last_status. The poll loop always sets it on every break path, so the initial `None` and the unwrap_or_else fallback were both dead. Refactored to `let after = loop { ...; break s; };`. cargo check is now clean. cargo test --workspace --bins: 614 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 15:34:02 -04:00
pub use prod_orchestrator::ProdContainerOrchestrator;
feat(container): ContainerOrchestrator trait, RpcHandler uses it in prod Step 4 of the rust-orchestrator migration. Unifies the container lifecycle surface behind a single trait so the RPC layer stops caring whether it is talking to the dev or prod orchestrator. * New trait core/archipelago/src/container/traits.rs: ContainerOrchestrator with install / start / stop / restart / remove / upgrade / status / list / logs / health, all keyed by app_id. Every method is async_trait-based. * ProdContainerOrchestrator: the lifecycle methods are moved from inherent impl into the trait impl (avoids name-shadowing recursion). Adoption and reconcile remain inherent since only main.rs / BootReconciler call them. * DevContainerOrchestrator: new trait impl that forwards to the existing Dev-named methods, applying the dev container-name + port-offset rules internally. New load_manifest_for() helper resolves app_id to <data_dir>/apps/<app_id>/manifest.yml so trait-level install(app_id) works in dev too. install_container(manifest, path) stays inherent for the manifest-path RPC shape. * RpcHandler now holds Option<Arc<dyn ContainerOrchestrator>> and, when in dev mode, a separate Option<Arc<DevContainerOrchestrator>> for the manifest_path install RPC. In prod mode RpcHandler::new() constructs a ProdContainerOrchestrator and calls load_manifests() at startup. * All seven container-* RPC guards no longer say dev mode required. container-install still requires dev mode because its manifest_path argument has no prod meaning; every other container RPC now works in both modes via the trait. BOOT STILL DOES NOT USE THIS. main.rs wire-up (Step 6) and BootReconciler (Step 5) come next. Until then the prod orchestrator is constructed but nothing populates /opt/archipelago/apps so it has zero manifests to manage, matching the pre-Step-4 behaviour. Verification: cargo build -p archipelago clean (11 expected unused method warnings for methods not yet wired from main.rs). cargo test -p archipelago: all 21 container::* tests pass (16 prod_orchestrator + 5 others). 24 other test failures are pre-existing and unrelated (identity_manager / session / wallet / mesh / credentials — all independently flaky on file-backed state).
2026-04-22 18:56:52 -04:00
pub use traits::ContainerOrchestrator;
Reference in New Issue Copy Permalink