archy/docs/UNIFIED-TASK-TRACKER.md

# Unified Task Tracker — OTA 1.8.0 + Master Plan

Single working list for everything left before 1.8.0 ships and the next master-plan
exit criteria (multinode + workstreams B/C/D) are met. Supersedes the open-task
sections of `docs/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md`
as the day-to-day tracker — those docs remain the historical record / detailed
narrative and are still linked from here where useful. **Ordered fastest/simplest
first** so we work top-down instead of hunting across docs.

Verified against actual code state on 2026-07-01 (not just doc text — several
items the source docs still listed as "open" turned out to already be shipped;
those are marked ✅ below with the commit that did it, so we stop re-litigating them).

---

## Tier 0 — Quick / mechanical, no blockers

- [ ] **Update `tests/lifecycle/TESTING.md`'s stale Release Gates checklist** (lines
      289–296) — several boxes are unchecked but actually true now:
  - #1 bitcoin-stops: covered by `tests/lifecycle/bats/bitcoin-knots.bats` stop/restart
    tier, included in the 5/5 green gate run.
  - #2 `ARCHY_ITERATIONS=5` on .228: **GREEN 2026-06-23 per CLAUDE.md** — check the box.
  - #5 cargo 0 warnings: confirmed 0 warnings on `cargo build --release` (2026-07-01).
  - #7 layman changelog: `CHANGELOG.md` is backfilled with layman-readable entries
    through v1.8.00-alpha — check the box.
  - Leave #3 (multinode), #4 (backend-survives-restart / Phase-3 default-on), #6
    (LoC decision), #8 (tag pushed) unchecked — genuinely still open, see Tier 2/3.
- [x] ~~Finish the archival/full-node manifest generalization~~ — investigated 2026-07-01:
      the hardcoded fallback names in `dependencies.rs:48-52` (`electrs`, `mempool-electrs`,
      `mempool-web`) are legacy **alias** ids for `electrumx`/`mempool`, resolved via
      id-mapping in a dozen other places (`install.rs`, `runtime.rs`, `config.rs`, etc.),
      not separate un-migrated apps with their own manifests. `electrumx` and `mempool`
      themselves already declare `bitcoin:archival`. The fallback is correct as-is —
      not tech debt, closing this item rather than risk breaking alias resolution.
- [x] ~~Confirm/close the Portainer image-pin item~~ — confirmed 2026-07-01:
      `146.59.87.168:3000/lfg2025/portainer:2.19.4` is present in `podman images` on
      all 3 LAN nodes (.116/.198/.228), i.e. actually resolvable/pulled from the mirror.
      Not a live bug.
- [x] ~~grafana Quadlet "stuck activating"~~ — checked live on .116 (2026-07-01):
      `grafana.service` is `active (running)`, container `Up 2 hours (healthy)`. The
      2026-06-21 report is stale for grafana. **strfry still unconfirmed** — not
      installed on any of .116/.198/.228 to check directly; low priority until someone
      actually needs it installed.

## Tier 1 — Medium effort, unblocked

- [x] ~~immich → Quadlet migration~~ — investigated 2026-07-01, turned out already done:
      immich uses the same `install_stack_via_orchestrator` primitive as netbird/btcpay
      (`immich_stack_app_ids()` in `stacks.rs:690`), and is confirmed running as real
      Quadlet units live on .228 (`immich_server.container`, `immich_postgres.container`,
      `immich_redis.container`, all active). Not a legacy in-cgroup app — the only
      remaining piece is the fleet-wide Phase-3 default-flip, already tracked in Tier 2.
- [x] ~~Netbird reinstall adoption path~~ — investigated 2026-07-01, **not a bug, by
      design.** `adopt_stack_if_exists()` (`stacks.rs:140-198`) is only used as a
      fallback when the orchestrator has no manifest for the app — there's nothing to
      render certs/config from in that case, so skipping rendering is correct. When
      the orchestrator *does* have the manifest (the normal path), the reconcile loop
      already re-renders certs even for adopted-running containers, fixed in
      `4519dbf0` (`prod_orchestrator.rs:1707-1708`).
- [x] ~~TanStack Query (or equivalent) investigation~~ — spike complete 2026-07-01,
      **recommendation: don't adopt / close as not needed.** Only 3 stores actually fetch
      data, WebSocket push already handles hot data (server-info/package-data), no
      cache-invalidation or stale-data bugs found, migration would touch 62 RPC call
      sites for no concrete payoff. If boilerplate ever bothers us, extract a
      `usePolling()` composable instead — much cheaper than a query-cache migration.

## Tier 2 — High effort, mostly unblocked (the actual next exit criteria)

- [~] **Multinode test pass** (`docs/multinode-testing-plan.md`) — worked the
      preconditions on .198 2026-07-01:
  - ✅ cleared 2 stale failed-unit records (`archy-mempool-db.service`,
    `meshtastic.service` — both `not-found`/dead since 6 and 5 days ago, harmless
    bookkeeping, `systemctl --user reset-failed`).
  - ✅ nginx `/app/lnd/` proxy target confirmed correct (→ `18083`, matches the
    running `archy-lnd-ui` port) — the plan's "stale proxy target" concern doesn't
    apply here.
  - ⛔ .198 disk (448GB) is below the 1TB archival threshold + was only 21%
    through IBD — user chose to **swap in a different node** rather than wait/add
    storage. **.116 ruled out** (no bitcoin container installed at all, just the
    UI companion). **.120 ruled out** (reserved for another developer). **.5**
    (archy-x250-beta, Tailscale `100.72.136.5`) chosen: also sub-1TB (472GB, so
    still pruned — that ceiling is shared by every non-.228 node), but **fully
    synced** (`ibd:false`, blocks==headers 956,240). Bootstrapped bats 1.11.1 +
    jq 1.7.1 onto it 2026-07-01 and **launched the 5× destructive gate
    (`ARCHY_ITERATIONS=5 ARCHY_ALLOW_DESTRUCTIVE=1`) — running now**, log at
    `/tmp/gate.log` on .5, background poller watching for the `RESULTS` banner.
  - Once .5's gate reports: bring the rest of the fleet to precondition, then the
    cross-node federation/mesh/transport suites. This is the literal
    "next exit criterion" called out in `CLAUDE.md`.
- [ ] **Phase-3 Quadlet default-flip** — code is validated + opt-in via
      `ARCHIPELAGO_USE_QUADLET_BACKENDS=true` on .228/.198 already (confirmed live
      2026-07-01). Ready to flip (`config.rs:256` + its test) the moment the .5 gate
      reports clean — deliberately NOT staged uncommitted in the tree (a prior attempt
      left an uncommitted flip sitting around and that caused confusion; it's a 2-line
      change, faster to just do it fresh once confirmed).
- [x] ~~Per-app test coverage for the ~30 apps with zero automated coverage~~ —
      **reframed 2026-07-01, mostly a non-issue.** `all-apps-matrix.bats` +
      `all-apps-lifecycle.bats` already give EVERY installed app generic baseline
      coverage (no stuck state, no error state, stop/start/restart survives, UI
      reachable). The real gap is narrower: **34 apps lack app-specific assertions**
      (health endpoints, API queryability, data integrity) beyond that baseline —
      aiui, bitcoin-core, botfights, core-lightning, did-wallet, fedimint-clientd,
      fedimint-gateway, fips-ui, gitea, grafana, home-assistant, indeedhub (+5
      sub-containers), jellyfin, lightning-stack, lnd-ui, morphos-server, netbird
      (+2 sub-containers), nextcloud, nostr-rs-relay, photoprism, portainer, router,
      searxng, strfry, uptime-kuma, vaultwarden. Not urgent — baseline coverage is
      real safety net; treat as a backlog "nice to harden further," not a gate item.
- [x] ~~Convert remaining multi-container legacy stacks to the manifest-owned model~~ —
      **investigated 2026-07-01, DONE, nothing left.** All 5 real multi-container
      stacks (btcpay, mempool, immich, netbird, indeedhub) are on the
      `install_stack_via_orchestrator` pattern (`stacks.rs`). saleor was removed from
      the codebase; portainer/home-assistant/grafana are single-container
      manifest-driven apps, never stacks; fedimint/fedimint-gateway/fedimint-clientd
      are 3 separate single-container apps with manifest dependency edges, not a
      coordinated stack. Workstream A's stack-migration tail is fully closed.
- [ ] **Developer tooling CLI suite** (validate/render/local-install/lifecycle-test) —
      APP-PACKAGING-MIGRATION-PLAN.md step 5, needed before external devs can publish.
- [~] **Cross-node federation/mesh/transport suites** — **big find 2026-07-01: these
      already exist**, just aren't wired into the gate or documented as existing:
      `tests/multinode/smoke.sh` (federation pairing/sync, FIPS anchor, peer content
      browse, tombstone-removal regression tests), `tests/multinode/meshtastic.sh`
      (8-stage on-air mesh test), harness in `tests/multinode/lib/multinode.bash`.
      **Actually ran `smoke.sh` live against .116↔.228 2026-07-01: 14 passed, 1
      failed, 1 skipped.** Confirms federation pairing (both directions), FIPS
      anchor connectivity (both nodes), and peer-content-browse-over-mesh (the
      v1.7.95 fix) all genuinely work node-to-node right now.
  - ⚠️ **Real robustness gap found**: `node_rpc()` in `tests/multinode/lib/multinode.bash`
    has no `--max-time` on its curl calls — a slow server-side RPC hangs the whole
    suite with zero feedback (this is what looked like a hang before it eventually
    completed on its own). Cheap fix, not yet applied.
  - 🐛 **Real regression found and root-caused**: removing a federation node
    (`federation.remove-node`) doesn't reliably stick — B reappeared in A's peer
    list after removal in the live test. Root cause: `remove_node()`
    (`core/archipelago/src/federation/storage.rs:187`) does
    `let _ = tombstone_did(data_dir, did).await` — **silently swallows the
    tombstone write's errors.** If that write fails (disk I/O, permission,
    transient issue), the peer is removed from `nodes.json` but never actually
    tombstoned, so the next background sync/notify-join re-adds it — the
    tombstone check at `handlers.rs:592-599` passes because the DID was never
    recorded as removed. Diagnosed as a **pre-existing logic gap**, not a fresh
    regression from the v1.7.95 fix. **Not fixed yet** — this is federation/trust
    code, deliberately not touching it blind; needs a careful fix (surface the
    tombstone-write failure instead of swallowing it, and/or retry) plus
    re-verification with `smoke.sh` before considering it closed.

## Tier 3 — Blocked on a decision or resource only you can supply

- [ ] **Version naming decision (1.7.99-alpha → 1.8.0 vs 1.8.00-alpha)** — code is
      otherwise ready to tag; this is a one-line decision, then a mechanical bump +
      tag + push. **Needs your call**, not more engineering.
- [ ] **Workstream B signing ceremony** — `core/archipelago/src/trust/anchor.rs:21`
      still has `RELEASE_ROOT_PUBKEY_HEX = None`. Needs the offline
      `RELEASE_MASTER_MNEMONIC` to run `docs/workstream-b-signing-runbook.md`'s
      4-step ceremony — can't be automated by me.
- [ ] **Bitcoin multi-version fleet-wide OTA** — `.228` fully working on branch,
      per your prior gating this rollout is explicitly held for your decision on
      timing (`docs/bitcoin-version-bulletproof-rollout.md`).
- [ ] **3ccc stock-Meshtastic RF validation** — needs a live send/receive test with
      physical radios in your hands; code fix is in place, just unverified live.

## Backlog — deferred, no scope decided, low priority

- [ ] **Marketplace protocol (workstream C)** — design-only (`docs/marketplace-protocol.md`),
      no tooling/trust UX built. Future work, not urgent.
- [ ] **DHT distribution (workstream D)** — confirmed design-only, no code
      (`docs/dht-distribution-design.md` explicitly says "Status: Design (no code yet)");
      an experimental iroh provider skeleton exists behind a feature flag for future
      PoC measurement, nothing fleet-facing.
- [ ] **Custom live voice-call protocol** — deprioritized 2026-07-01 per user request;
      scope not yet decided. Revisit after the tiers above are worked down.

---

*Historical narrative and detailed per-session logs remain in
`docs/SESSION-1.8.0-OTA-PROGRESS.md` and `docs/PRODUCTION-MASTER-PLAN.md` §6/§8b —
this doc is the live "what's left, in priority order" list. Update it (don't just
append to the old docs) as items close or new ones surface.*