archy/docs/release-from-latest-main.md

Release OTA and ISO from latest main

This is the operator handoff for producing a release from whatever commit is the reviewed tip of main at the time. A release is not ready merely because the artifacts build. Every active regression, live-node acceptance, mirror and publication gate below must pass.

1. Freeze an exact reviewed source revision

Use a fresh clone or clean worktree. Do not build from an UAT node or a dirty tree.

git switch main
git pull --ff-only ngit main
git status --short
git rev-parse HEAD
python3 scripts/check-git-mirrors.py --local --all

git status --short must be empty. The full mirror audit must show matching advertised branches/tags on ngit and Gitea. Inventory and resolve any unrelated drift; never force-push, delete refs, or rewrite published history merely to make the check green. Record the accepted ngit proposal and merge commit in the release acceptance ledger.

Read and retain every unchecked item in:

• docs/post-1.8.22-regressions-20261001.md
• the current release acceptance ledger and UAT checklist

Run the source gates and the documented live-node matrix. On a node with installed apps, backend tests must run only through the isolation wrapper:

scripts/test-backend-isolated.sh
tests/release/run.sh
python3 scripts/check-app-catalog-drift.py --release --strict

Keep source/unit results separate from actual-node acceptance. In particular, prove that paid-file recovery never makes a second payment and that app cleanup preserves wallets, persistent data, and uninstall decisions.

2. Prepare and sign the OTA release

Choose a new SemVer that has never been published. Do not move or replace an existing release tag. Curate the new top entry in CHANGELOG.md, then preview:

release_version=X.Y.Z-alpha
scripts/create-release.sh "$release_version" --dry-run

When all gates and review are complete, run the real preparation from clean main in an interactive terminal:

scripts/create-release.sh "$release_version"

The script builds the backend, frontend, AIUI and radio tools, creates a signed pending manifest and staged OTA artifacts, commits the release preparation, and creates annotated tag v$release_version. During the signing prompt, paste the 24-word release master mnemonic once, press Enter, then Ctrl-D on the next line. Never put the mnemonic in a command, file, chat, log, or Git. Do not use RELEASE_MASTER_MNEMONIC for the normal operator ceremony.

If an already-prepared pending manifest needs signing on the offline/operator terminal, first ensure the release binary was built from the exact frozen commit, then run:

bash scripts/sign-manifest.sh \
  "releases/pending/v${release_version}/manifest.json"

The script cryptographically verifies the result against the public release root pinned in the binary. A failed verification is a hard stop.

Review the preparation commit, tag target, staged artifacts and signature. Push the exact reviewed commit/tag to both publication mirrors according to the ngit-first workflow; do not create independent merges on each platform.

3. Build and sign the installer ISO

The ISO builder requires clean main, matching versions, the annotated tag and the signed live manifest. After OTA publication has safely promoted the live manifest (next section), return to the exact tagged clean tree and run:

scripts/build-iso-release.sh

For a normal release, do not pass --skip-gates or --no-qemu. The command runs the release harness, strict catalog check, artifact checks, ISO build, mount-level smoke test and headless QEMU boot. Keep its final PASS summary as release evidence.

Sign the generated ISO checksum document in the operator ceremony:

iso_path=/absolute/path/to/archipelago-${release_version}.iso
bash scripts/sign-iso-checksums.sh "$iso_path"
core/target/release/archipelago ceremony verify \
  "$iso_path.sha256.json"
sha256sum -c "$iso_path.sha256"

The signing script again expects one mnemonic paste, Enter, then Ctrl-D. It does not place the mnemonic on disk.

4. Publish without exposing incomplete updates

Configure the Gitea remote with a public HTTPS URL and keep its credentials in Git's credential helper, never in the remote URL. Then run:

scripts/publish-release-assets.sh "$release_version" gitea-vps2

This script deliberately pushes the tag first, creates the release, uploads and byte-verifies the OTA assets, and pushes the manifest-bearing main last. When the signed ISO files exist, the same command attaches and verifies them. Never manually publish releases/manifest.json before its referenced assets are downloadable and verified.

After publishing, mirror the exact accepted main commit and annotated tag to ngit and Gitea, then verify both:

python3 scripts/check-git-mirrors.py --local \
  --ref "refs/tags/v${release_version}"
python3 scripts/check-git-mirrors.py --local --all
scripts/check-release-assets.sh releases/manifest.json

A failed push, missing ref, tag-object mismatch, unavailable mirror, incomplete asset, or mismatched byte hash blocks publication. A main-only check is not full historical mirror parity.

5. Canary and final acceptance

Before announcing the release:

1. Apply OTA to one non-critical canary from the signed manifest.
2. Reboot it and verify management health, signer/native identity, installed and deliberately removed apps, wallet state, persistent app data, terminal, networking/FIPS, and the active regression checklist.
3. Exercise rollback/recovery without changing wallets or app data.
4. Boot the ISO in QEMU and on representative physical hardware; perform a clean install and an upgrade-path check, then repeat the same acceptance matrix.
5. Verify public release downloads and both Git mirrors once more. Record exact commit, annotated tag object, artifact hashes, signer verification, node evidence, ngit proposal/merge, known deferrals and operator acceptance in the release ledger.

Do not call the release complete if any required live acceptance, signature, mirror, asset, OTA reboot/rollback, or ISO boot/install result is missing.