archy/docs/managed-update-recovery-implementation.md

Managed update runtime recovery

Status: integrated candidate; combined Rust suite and real Podman/systemd recovery primitives pass. Full application cutover qualification is pending. No live update, snapshot, stop, backup or rollback has been performed by this work. Active deployed source is unchanged.

The managed path captures original source Quadlet bytes, mode, immutable image, container identity, launch configuration and running intent. It requires an original-hash-bound reviewed forward plan and verifies every planned image against the already prepared catalog image. New manifest configuration/hooks are applied on the forward path; rollback uses the captured original recipe and a private, local-only writable-layer image. AutoRemove rollback recreates containers and does not claim to restore their original IDs. Stopped supervised stacks currently fail before mutation; the retained-container and separate stopped-stage paths cover only their respective supported cases.

The legacy IndeeHub controller runs under the same inherited lifecycle lock and operation-owned reconciliation holds. Original writable layers are captured before any destructive stop. The controller fences ingress, drains supported legacy work, takes coherent quiescent volume/database backups and retains the fence through cutover or recovery. Its exact source hash must match the installed script. Current binary startup promotes its embedded controller before recovery/reconciliation, including over an older runtime payload; the updater still verifies the exact on-disk hash.

Completed updates and verified runtime restorations publish exact unit recipes before releasing holds. Routine drift reconciliation validates those recipes; it does not regenerate them from a newer catalog. Catalog-driven pre-start file and mount mutations are skipped for these pinned installations. Missing saved units/images are explicit recovery failures, never permission to reconstruct a different runtime. Explicit uninstall removes the installed recipe, and completed old journals cannot recreate it. A new reviewed transaction replaces the recipe.

Opted-in API registration environments must match an already provisioned pin and the existing node identity in both manifest and exact Quadlet. Administrative plan preparation must use the existing installer provisioning code. Execution never invents a node identity or accepts a browser-supplied unit or hook.

Runtime restoration does not establish database compatibility. The controller's restored-release verifier compares original table schemas and row commitments, permitting only the exact reviewed additive migration prefix and empty new application tables. Any other data/schema change keeps ingress closed. This is not automatic database rollback or a promise that arbitrary migrations are reversible.

Qualification required before integration/activation:

• Isolated backend compile and injectable lifecycle/fault tests, including lost replies, daemon interruption, foreign units/holds and preflight failures.
• Disposable real Podman/systemd and PostgreSQL execution of the controller and adapter. Sixteen pure controller tests currently pass; SQL/runtime behavior is not yet qualified.
• Final seven-member private plan with installer-resolved identity environment, verified local images and source-unit provenance; review required changes and retained operator configuration without exposing secret values.
• Disk-capacity and recovery-image retention checks, backup integrity and a documented recovery path for missing runtime artifacts.
• Actual-node controlled deployment and acceptance, preserving persistent data.

Resumed qualification — 2026-10-07

Backup verification now requires the full database/four-volume artifact set and rechecks SHA256, including same-size corruption. Nineteen pure controller tests pass. The owned, network-none PostgreSQL fixture passes unchanged/additive commitments, rejects four data/schema/history mutations, restores a real custom dump with matching original commitments, and rejects a truncated dump. It mounts no live volume and removes only its own container. This does not qualify actual application writer drain or the complete supervised systemd cutover.

The updater compiled and its full isolated suite ran: 1,958 passed, one failed, five ignored. The failure is the old snake_case rental receipt JSON fixture; c2c4d915 already corrects that exact test on the release candidate branch. Do not duplicate or suppress it here. Integrate and rerun the complete candidate suite before claiming a green backend gate. The earlier interrupted compile and PostgreSQL timeout remain failed/incomplete attempts, not acceptance.

Evidence: /tmp/archy-resumed-20261007-updater-full-backend.log, /tmp/archy-resumed-20261007-indeehub-controller-tests-final.log, and /tmp/archy-resumed-20261007-indeehub-postgres-restore.log.

Integrated candidate checkpoint — 2026-10-07

Local integration at 7fb7ee80 passes the complete isolated backend suite: 2,000 passed, zero failed, five ignored. The stale receipt fixture failure above is resolved by the already-integrated correction. Disposable real Quadlet AutoRemove recovery primitives pass with injected target failure, original writable-layer/configuration restoration and unchanged persistent fixture bytes. Repeatable fixture: tests/lifecycle/supervised-runtime-primitives.py. This does not qualify the complete seven-member app drain/cutover adapter.

The fresh-backup restore barrier is being added after this checkpoint; its qualification and new embedded-controller build remain separate from these previously passing results. No live IndeeHub deployment has been changed.

Fresh database backup restoration now runs through the controller's production method in a disposable network-none PostgreSQL with no external mounts or published ports. The original local image is pinned, dump restore must exactly match captured database commitments, and durable proof binds the operation, image, dump hash and baseline. Ownership-checked cleanup survives retry and refuses foreign fixtures. Verification rejects a missing/stale restore proof.

All21 pure controller tests pass. The real PostgreSQL fixture passes valid restoration, rejects truncated and wrong-database dumps, checks cleanup and retained admission on failure, and retains the four prior mutation rejection checks. Evidence: /tmp/archy-20261007-fresh-backup-restore.log. Final real restore-barrier checks pass on PostgreSQL15.17 and16.13 after waiting for the final TCP server instead of the temporary Unix-socket bootstrap server. The initial PG15 restore failure is retained as failed evidence; its private command stderr was removed by fixture cleanup, so no exact cause is claimed. The stale backend compile was interrupted after the readiness edit; a fresh backend build/suite remains required for the final embedded controller. Volume-archive restore and the full supervised app cutover remain open gates.

Four-volume restore barrier — 2026-10-07

The controller now restores each fresh volume archive into separate owned storage before target startup. It rejects unsafe paths/links and unsupported special files, checks restored bytes, links, ownership and modes, and rearchives the restored tree to compare ACLs and extended attributes explicitly (GNU tar compare alone does not check xattrs). Durable proof binds all four archive hashes to the operation. Failure retains ingress and lifecycle holds; retry removes only the operation-owned fixture. No production volume is mounted or modified by restore verification.

All24 pure controller tests pass. The real rootless fixture passes four archives containing hidden files, hardlinks, symlinks, mapped numeric ownership, ACLs and xattrs; changed restored bytes/xattrs and unreadable archives are rejected. Evidence: /tmp/archy-20261007-volume-controller-tests.log and /tmp/archy-20261007-volume-restore.log. Repeatable fixture: tests/regression/test_indeehub_maintenance_volumes.py. Full seven-member application cutover and final backend build remain separate gates.

Explicit legacy preparation commands

The candidate adds two offline administration commands; neither starts the daemon, changes a catalog, enables registration/publication, or starts/stops an app:

• archipelago prepare-indeehub-registration DATA_DIR MANIFEST_JSON PRIVATE_OUTPUT_JSON validates a private opted-in API manifest with an immutable image and both feature flags disabled, loads the existing node identity, invokes the existing installer pin provisioner, and writes a private resolved manifest. Retry preserves the same audience and identity; missing identity is an error, never identity generation.
• archipelago prepare-indeehub-update DATA_DIR validates the exact seven-member installed reviewed plan, original unit hashes, local target images and registration pins under the lifecycle lock. It preserves observed original unit recipes before a newer catalog can drift-reconcile them. This records original installation evidence, not a fabricated completed update. Explicit uninstall removes those recipes through the existing path. Prepare the complete reviewed plan first and retain the current catalog until this command succeeds for all seven members.

Binary startup now promotes its exact embedded maintenance controller before recovery/reconciliation, including when an older dashboard payload is installed. The updater still checks the on-disk helper hash against the binary. These source changes passed full isolated backend qualification below; they have not been applied to Yaya. The full adapter fixture is prepared in a separate outbound-isolated QEMU copy-on-write VM, using public base images, the verified tracked baseline catalog and newly generated fixture credentials; no live app metadata/data is copied.

Preparation qualification: the final combined isolated backend suite passes 2,003 tests, zero failures, five ignored. Installer preparation preserves the existing identity/audience on retry and refuses absent identity or enabled feature flags. Original-recipe tests cover all-member preflight, retry, foreign edit preservation and explicit uninstall. An initial run had2,002 passes and one failure in the new fixture's assertion that the journal directory was absent; the shared readiness check creates an empty directory. The corrected test requires no journal files, which verifies the intended absence of a fabricated completed transaction. Evidence: /tmp/archy-20261007-indeehub-final-backend-recheck.log; initial failed fixture evidence remains in /tmp/archy-20261007-indeehub-final-backend.log. Optimized artifact build and full real VM adapter acceptance remain pending.

Full RPC dispatch correction — 2026-10-07

Tracing the actual package.update path found that IndeeHub was marked as a stack but missing from the pinned stack-image mapping. It therefore resolved only its frontend before the seven-member runtime membership check. The mapping now covers PostgreSQL, Redis, MinIO, relay, API, worker and frontend in dependency order.

Managed preflight now resolves the signed catalog references locally against the reviewed immutable plan and checks original unit hashes/registration pins before image preparation. Only verified local digest references reach preparation, so unchanged mutable dependency tags cannot trigger a registry pull before the private plan is checked. Unmanaged updates retain registry preparation. Inventory and plan refusals clear progress and the inner Updating state; the asynchronous RPC wrapper also retains its existing failure/scanner cleanup.

The final source passes 2,004 isolated backend tests, zero failures, five ignored: /tmp/archy-20261007-indeehub-dispatch-final-backend.log. A prior 2,004-test receipt predated the last preparation-state cleanup and is not the final-byte result. The optimized build was deliberately interrupted after these integration gaps were found. A test-profile application executable is being built solely for the isolated full RPC rehearsal; final release optimization/deployment remains pending that rehearsal. No live IndeeHub stack or catalog has been changed.

Manager startup guard found by real VM rehearsal — 2026-10-07

The isolated VM imported all seven public baseline images and started a fresh synthetic stack with generated credentials, internal-only networking and retained unit recipes. Offline original-recipe preparation succeeded. Starting the real manager then exposed a destructive ordering bug before the first update RPC: ordinary environment-drift reconciliation stopped/removed original members before install_fresh reached its existing managed-recipe refusal. The journal records this sequence for worker, API, PostgreSQL and MinIO. This was not a demonstrated OOM: guest kernel records showed no OOM and its disk-backed swap was available.

The new guard precedes staged stops, secrets, hooks, dynamic configuration, drift repair and generic recreate paths. It preserves explicit stop/uninstall markers, validates saved unit bytes/mode, and observes an already-running managed runtime. Missing or stopped managed runtimes refuse automatic repair; explicit owner Start/Stop/Restart uses the exact saved systemd unit after validation. Restart preflights all member units/images before reverse dependency stop and dependency start order. Managed members skip legacy network, catalog-port repair and raw Podman fallback. Held update records refuse these owner lifecycle mutations. The outer ownership sweep also excludes saved/held members, and generic staged cleanup refuses a saved member before disabling/removing its unit. Regression scenarios verify unchanged runtime inventory and observation-only calls for running, stopped, missing and modified-unit cases, plus durable stop/uninstall choices. The health monitor also refuses automatic restart for saved, held or damaged recovery evidence and reports the unhealthy app without mutating it. Updating packages are excluded from health recovery. Combined isolated validation is pending: the superseded compile was deliberately stopped before any tests ran so these final lifecycle/health changes can share one full suite.

The VM is now off. SSH shutdown attempts timed out and ACPI powerdown did not complete; QMP quit terminated only this transaction-idle synthetic fixture to release its RAM during compilation. The next dedicated 4 GB boot must mask the old manager in GRUB, check guest filesystem recovery, install the corrected binary and recapture baseline before acceptance. Failed-run identities are retained in the VM's indeehub-fixture/startup-drift-failure-evidence; all seven exact saved units were verified and a new synthetic baseline explicitly recaptured. This recapture is not rollback acceptance. The first RPC script stopped at its seven-running precondition, so no missing-plan, rollback or successful update RPC acceptance has yet completed. No live Yaya stack, catalog, wallet or payment was changed.

Hardened-controller context correction (2026-10-07)

The final lifecycle/payment isolated suite passed 2,006 tests (zero failures, five existing ignores), with all 532 captured inputs unchanged. Receipt: /tmp/archy-paid-indee-lifecycle-backend-20261007.log. This result predates the controller-launch change described below; it is not a current-source full pass.

An isolated VM probe reproduced another actual integration defect before rebuilding its executable. Plain system-service podman exec /bin/true passed, but matching the manager's ProtectSystem=strict, Delegate=yes and writable-path policy made API and PostgreSQL exec fail with exit126. Both commands passed in a user systemd scope. The scope retained the same inherited flock open description: inode check and nonblocking exclusive re-lock both passed while the parent held the lock. All seven container IDs remained unchanged by these probes. Private guest receipt: /home/archipelago/indeehub-fixture/context-probe.receipt.json; bounded probe source: /tmp/indeehub-vm-context-probe.py on the development host.

The controller now launches through systemd-run --user --scope --quiet --collect before the pinned Python helper. This preserves synchronous pipes and inherited lifecycle locking while executing rootless Podman from the user unit context. Rust syntax and diff checks pass. Corrected executable and actual full transaction qualification are still required; no live IndeeHub activation is claimed.

The fixture's earlier forced shutdown required root journal recovery and rebuilding seven stale synthetic Podman runtime records from their byte-verified saved units; their private pre-recovery inspection was retained. These recreated IDs must be recaptured as a new baseline, never described as preserved across that shutdown. A direct-kernel initramfs-only recovery boot installed an absent-marker manager startup condition before normal boot, and ConditionResult=no verified the old manager never started. The subsequent diagnostic boot shut down gracefully.

Actual RPC preflight and nginx namespace evidence (2026-10-08 UTC)

Corrected VM executable built successfully from frozen inputs (normal binary SHA256 3cbe5a5c3a74e6463ab0874c74e23dfca68f0bfc3fe54883f0edf69abb37ac0d; stripped transfer SHA256 753f8ba6ac342c8f4b9a19c8079a51dfd1da4dcb517d4ea4d0e54035c22f5788). Receipt: /tmp/archy-indeehub-corrected-executable-20261007.json. It predates the later rental guard and nginx helper correction; not a release artifact.

Actual manager startup completed a62-app reconcile pass with all seven IndeeHub members NoOp, every baseline container identity preserved, and API/manager HTTP200. Subsequent real update attempts initially met the legitimate background lifecycle lock. A temporary behavior-preserving fixture tracer confirmed later admission succeeded. One early diagnostic teardown interrupted asynchronous preflight and is invalid as source-defect evidence. The corrected diagnostic waited for its terminal refusal before restoring the deliberately withheld plan and removing the tracer; all seven original identities remained, no supervised journal existed. No speculative lifecycle-lock patch was made.

The admitted reviewed-plan RPC then reached Prepared → Editing → Restoring. All seven recovery images exist; target startup never began. The controller remained Prepared because sudo nginx -T attempted to open /run/nginx.pid inside the manager's read-only mount namespace. Holds and the unresolved journal were preserved; this is not a successful rollback or migration receipt.

A manager-equivalent hardened VM probe passed with the fixed command sudo -n /usr/bin/systemd-run --quiet --wait --pipe --collect -- /usr/sbin/nginx -T. It validated the required ingress guards without printing effective configuration or widening manager write access. The controller uses that fixed command now; 25 pure controller tests pass, including refusal before fence creation on dump failure. A new helper hash requires a matching backend rebuild. Candidate-helper qualification remains separate from actual backend transaction acceptance.

Legacy worker termination qualification — 2026-10-07

The held disposable-VM operation 4320fe90-8ab5-4496-a4e7-cf11ca0fd376 exposed the original worker's Node-as-PID1 SIGTERM behavior. A no-network, no-volume probe from its exact recovery image exited 137 without init and 143 with init. This is not graceful shutdown or proof of completed jobs.

The controller now requires an exact six-field nonnegative integer queue observation; absent active can no longer imply idle. Its narrowly scoped legacy worker path records forced idle termination, with graceful=false and completed_work_claim=false, only after paused all-zero observations before and after, closed ingress/frontend, exact operation/original/recovery image and known command, unchanged saved unit, original stop intent and died event, and proof the process is dead. Other writers' 137 exits remain refused. All 31 pure controller regressions passed, including missing/nonzero counts, reopened queue, changed identity, command, unit and process state. Actual worker-only classification passed under the hardened service → user scope with the lifecycle lock held.

The same fixture's API has npm as PID1 and one direct node dist/main child. After fresh empty-business-state proof and durable stop intent, an exact-command, parent-validated SIGTERM to that child stopped the original container; npm emitted exit 1. The existing clean-exit gate correctly retained the hold. API termination classification is still under review; no generic exit-1 allowance was added. Frontend/worker are confirmed stopped; API is stopped but unconfirmed; storage members remain running. Backup/fresh-restore, full RPC rollback/success and live activation are not passed. Installed pinned helper remains unchanged; this is private candidate-helper qualification only. Live Yaya remains unchanged.

Future worker image source now includes idempotent SIGTERM/SIGINT shutdown in app commit 29627fc with four passing Jest tests. Its image has not been built; the previous frontend/API-only candidate catalog cannot cover that new worker image.

API child-signal compatibility — 2026-10-07

The original API wraps node dist/main in npm PID1. Directly signalling that exact child produces npm exit 1, not a clean exit. The candidate controller now records this only as non-graceful empty-business termination, never as completed work. It requires the exact original/recovery image and saved unit, automatic restart policy no, fresh complete zero business-table/transaction counts, a paused empty queue, and durable operation/container/parent/child PID+starttime+command signal intent followed by syscall acknowledgement. Missing acknowledgement retains the hold; retries cannot infer one. Extra direct children, reused process identity, partial proof, OOM, forced exit 137, or replacement API writers are refused.

Post-stop queue verification now uses atomic, read-only Redis Lua, authenticated through stdin rather than secret command arguments. The observed original API queue endpoint must bind to the exact original Redis ID/image and shared network alias. All six counts, prioritized and waiting-children must remain zero, with admission paused. No default or absent field can establish empty state.

All 38 pure controller tests passed, including actual Node execution of the process selector. The actual hardened VM Redis observer passed. A uniquely named recovery-image API process probe, with no persistent mounts or published ports, passed exact child-starttime/signal-acknowledgement checks and exited 1 without OOM; bound Redis observations before/after were paused and entirely zero. The probe was removed. Its receipt is api-process-probe.receipt.json in the private VM fixture directory. This qualifies the primitive, not a full backend update. The earlier held API diagnostic lacks the new durable proof and remains unaccepted; it must be restored by the matching rebuilt manager before a fresh full update/rollback rehearsal. No live Yaya mutation or activation occurred.