archy/docs/registry-manifest-design.md

Registry-Distributed App Manifests — Design

Status: implemented — Phases 1–3 shipped (schema + catalog-wins overlay, signed publisher generator with embedded manifests for all apps, immich end-to-end via install_stack_via_orchestrator); Phases 4–5 (build-context apps content-addressed, drop apps/ from OTA) remain open. Updated 2026-07-08. Goal (north-star): every app installs from a manifest distributed via the signed app-catalog on the registry — no OS-level code reliance, no OTA-shipped disk manifest required. Rootless, signed, robust, reboot-survivable.

See also: docs/dht-distribution-design.md (this is its "discovery/authenticity" layer), MEMORY → project_manifest_driven_north_star.

---

1. Where we are today

Two distinct mechanisms, only one of which is registry-distributed:

Thing	Source	Reaches node via	Carries
apps/*/manifest.yml (48)	repo working tree	OTA: self-update.sh rsyncs apps/ → /opt/archipelago/apps/	full manifest (the orchestrator's real source of truth)
app-catalog.json (28)	releases/app-catalog.json	registry HTTP fetch, hourly, signed (app_catalog::refresh_catalog)	version + image override only

• Orchestrator registry = in-memory state.manifests: HashMap<app_id, LoadedManifest>, populated by ProdContainerOrchestrator::load_manifests() walking the disk dir. install(app_id) → loaded(app_id) → "unknown app_id" if absent.
• app_catalog.rs is already: signed (release-root, trust::verify_detached over the raw JSON), mirror-derived URLs, atomic cache at <data_dir>/app-catalog.json, forward-compatible (no deny_unknown_fields — adding fields never breaks old nodes).

Gap: the manifest itself is never registry-distributed. Every app — btcpay, grafana, immich — depends on an OTA-shipped disk file. That is the OS-level reliance to eliminate.

2. Target

The signed catalog entry carries the full manifest. The orchestrator loads manifests from the catalog cache (origin), falling back to disk only during the migration window. Publishing an app = editing the catalog + signing + push — no binary OTA, no disk manifest.

publisher: apps/*/manifest.yml  ──generate──▶  releases/app-catalog.json (embeds + signs)
node:      refresh_catalog() ──fetch+verify──▶ <data_dir>/app-catalog.json
           load_manifests()  ──merge──▶ state.manifests   (catalog wins; disk = fallback)
           install(app_id)   ──▶ render Quadlet unit (rootless, systemd-managed)

3. Schema change (app_catalog::AppCatalogEntry)

Add one optional, forward-compatible field:

/// Full app manifest, embedded so the app installs from the registry alone
/// (no OTA-shipped disk file). Carried as the raw value the publisher signed;
/// deserialized into `AppManifest` at load time. Absent during migration =>
/// the node uses the disk manifest fallback.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub manifest: Option<serde_json::Value>,

Why serde_json::Value, not AppManifest:

• keeps the signed preimage intact (we verify over the raw JSON bytes; a typed round-trip could drop/reorder unknown fields and break the signature),
• decouples catalog schema from manifest schema churn,
• deserialize + validate() happens at orchestrator load, exactly like from_file.

Authenticity is free: fetch_one already verifies the release-root signature over the whole document, so an embedded manifest is covered by the same signature. A present-but-bad signature is already a hard reject.

4. Orchestrator load path (load_manifests)

Extend (not replace) the disk walk:

1. Load disk manifests as today → disk: HashMap<app_id, LoadedManifest>.
2. Load catalog manifests from the cache: for each entry with manifest: Some(v), serde_json::from_value::<AppManifest>(v) then validate(); on success build a LoadedManifest { manifest, manifest_dir }.
3. Merge, catalog-wins: a catalog manifest overrides the disk one for the same app_id. Disk remains the fallback for apps the catalog doesn't cover (migration).
   • Rationale: the registry is the authoritative origin; disk is the legacy transport we're retiring. This matches app_catalog's "catalog verdict is authoritative when it covers the app" posture.
4. A catalog manifest that fails parse/validate is logged and skipped → disk fallback used (one bad entry never blocks the fleet, same as the disk walk).

manifest_dir for registry manifests — IMPLEMENTED

LoadedManifest.manifest_dir is used only in the ResolvedSource::Build branch (relative container.build.context resolution — two call sites). Image-only apps (ResolvedSource::Pull) never read it.

Decision (phase 1, shipped): keep manifest_dir: PathBuf (no Option ripple through the codebase). A catalog manifest with a build source is skipped so its disk manifest stays in effect — build contexts aren't registry-distributed until a later phase (content-addressed, per the DHT plan). For an accepted (image-only) catalog manifest, manifest_dir = the disk app dir if the app also exists on disk, else a sentinel <manifests_dir>/<app_id> (never read for image-only apps).

This is enforced by catalog_manifest_to_overlay(app_id, value) -> Option<AppManifest> in prod_orchestrator.rs, which returns None (→ disk fallback) for: unparseable value, embedded-id ≠ catalog-key, failed validate(), or a build source.

5. Publishing (publish-side generator)

Add a generator (extend create-release.sh / a small scripts/gen-app-catalog):

• walk apps/*/manifest.yml, parse, embed each as the entry's manifest (JSON),
• keep version/image/images derived from the manifest for the badge path,
• write releases/app-catalog.json, then sign with the existing release-root ceremony (archipelago ceremony / Phase 0 seed). Unsigned still accepted in the migration window.

6. Migration & rollback

• Backward compatible: old nodes ignore the new manifest field (no deny_unknown_fields) and keep using disk manifests.
• Forward: new nodes prefer catalog manifests, disk as fallback. Once the catalog covers every app and is verified live, drop apps/ from the OTA rsync.
• Rollback: delete <data_dir>/app-catalog.json (or revert the published catalog) → nodes fall back to disk manifests. No data touched.

7. Phases

1. ✅ Schema + load merge (this design): manifest field, load_manifests catalog-wins merge, unit tests (catalog overrides disk; bad catalog manifest → disk fallback; absent → disk); manifest_dir stayed a plain PathBuf (see §4). Image-only apps.
2. ✅ Publisher generator + signing: releases/app-catalog.json embeds a full manifest block per app (all disk manifests covered) and is verified via the release-root detached signature.
3. ✅ First real app end-to-end: immich installs via install_stack_via_orchestrator with generated_secrets; the install_immich_stack name survives only as an orchestrator-first wrapper.
4. ⏳ Build-context apps: content-addressed build contexts in the catalog (DHT swarm fetch) so companions stop needing disk too.
5. ⏳ Drop apps/ from OTA once coverage + live verification complete.

8. Open questions

• Do we embed manifests inline or reference them by content hash (BLAKE3) with a separate signed blob? Inline is simplest for Phase 1; hashing aligns with the DHT image-by-digest plan and keeps the catalog small. Lean inline now, revisit at Phase 4 when build contexts (large) need addressing anyway.
• generated_files with inline content (vs. source-dir) — already supported in the manifest schema? If so, registry manifests can carry small rendered files inline, removing another disk dependency.