archy/.planning/phases/09-botfights-platform-upgrade/09-04-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	requirements-completed	coverage	duration	completed	status
09-botfights-platform-upgrade	04	infra	docker vps2 botfights sqlite jwt gitea-registry		Running canonical BotFights arena container on VPS2 (146.59.87.168:9100) docker-compose.arena.yml — reproducible arena compose definition, registry image only docs/arena-deployment.md — full runbook (secret handling, seed method, TLS upgrade path, teardown) Canonical arena URL for downstream plans: http://146.59.87.168:9100	09-05 09-06 09-07	added patterns Read-only sqlite VACUUM INTO export for live-DB seeding without touching the source docker-compose healthcheck override for images built before the Dockerfile HEALTHCHECK existed	created modified /home/archipelago/Projects/botfight/docker-compose.arena.yml /home/archipelago/Projects/botfight/docs/arena-deployment.md	Canonical arena URL is http://146.59.87.168:9100 — plain HTTP, no DNS/TLS (user decision 2026-07-30, resolved before this plan ran) Data seed: full copy of archi-dev-box's botfights.db (115 bots / 102,440 fights) — user decision, option-c Image ref uses localhost:3000/lfg2025/botfights (not the public 146.59.87.168:3000) — Docker trusts loopback registries without insecure-registry config, verified working on VPS2 Added an explicit docker-compose healthcheck override since the running 1.1.0 image tag predates the Dockerfile's HEALTHCHECK directive	BOT-03	id description requirement verification human_judgment D1 Canonical BotFights arena container running and healthy on VPS2 at http://146.59.87.168:9100 BOT-03 kind ref status other ssh debian@146.59.87.168 'curl -fsS http://127.0.0.1:9100/api/health' — {"status":"ok","name":"botfights"} pass kind ref status other curl -fsS --max-time 10 http://146.59.87.168:9100/api/health (off-host, from archi-dev-box) pass kind ref status other docker ps --filter name=botfights-arena --format '{{.Status}}' -> Up ... (healthy) pass false id description requirement verification human_judgment D2 docker-compose.arena.yml committed — registry image only, no build, no payment env, no TRUSTED_PROXY BOT-03 kind ref status other grep -c 'BOTFIGHTS_WALLET_ENCRYPTION_KEY|BOTFIGHTS_NWC_URL|BOTFIGHTS_CASHU_MINT_URL' docker-compose.arena.yml — only commented lines pass kind ref status other docker inspect botfights-arena env — no ARENA_UPSTREAM_URL, no TRUSTED_PROXY entries pass false id description requirement verification human_judgment D3 docs/arena-deployment.md runbook committed with host/port/secret-handling/seed-decision/NPM-TLS-upgrade-path/teardown, including the arena-as-relay architecture section BOT-03 kind ref status other test -f docs/arena-deployment.md; grep -c 146.59.87.168 docs/arena-deployment.md -> 15 matches pass false id description requirement verification human_judgment D4 Full-copy data seed live on the arena (115 bots via read-only export from archi-dev-box); source database provably untouched BOT-03 kind ref status other curl http://146.59.87.168:9100/api/bots -> 100 rows (default filter excludes botType=classic); GET /api/bots?type=classic -> 15 rows; 100+15=115 matches source pass kind ref status other stat mtime/size of /var/lib/archipelago/botfights/botfights.db before and after export — 1782916151 / 367144960 bytes, byte-identical pass false	~65min	2026-07-31	complete

Phase 9 Plan 4: Canonical BotFights Arena on VPS2 Summary

One canonical BotFights arena deployed and verified on VPS2 at http://146.59.87.168:9100 (plain HTTP, no DNS/TLS by user decision), seeded with a full read-only copy of archi-dev-box's 351 MB production database (115 bots, 102,440 fights) — the shared endpoint every node's BotFights instance will proxy to.

Performance

• Duration: ~65 min
• Completed: 2026-07-31T01:25Z
• Tasks: 2 of 3 executed (Task 1 was a decision checkpoint already resolved before this run — see plan frontmatter/resume-signal)
• Files modified: 2 (both new)

Accomplishments

• Deployed botfights-arena container on VPS2 running the published localhost:3000/lfg2025/botfights:1.1.0 registry image (no local build), bound to 9100:9100, with a host-generated JWT_SECRET (0600 .env, never committed), no TRUSTED_PROXY (direct exposure, rate-limits on real socket peer IP), and no payment env vars.
• Exported archi-dev-box's live botfights.db (351 MB, 115 bots, 102,440 fights) via a read-only SQLite URI connection (mode=ro) + VACUUM INTO, confirmed the source file's mtime/size were byte-identical before and after (1782916151 / 367144960), and seeded the arena's volume with the full copy.
• Verified the arena healthy both on-host (127.0.0.1:9100/api/health) and off-host (146.59.87.168:9100/api/health from archi-dev-box) — internet-reachable with no firewall changes needed (host firewall already permitted the port; ufw isn't even installed on this host).
• Committed docker-compose.arena.yml and docs/arena-deployment.md to the botfight repo (staged by explicit path — the working tree is shared with another concurrently-active agent) and pushed to origin/main.

Task Commits

1. Task 2/3 combined (deploy + prove internet-reachable) — botfight repo e4b82fd: feat(09-04): deploy canonical BotFights arena on VPS2

Plan metadata: this SUMMARY + STATE/ROADMAP updates, committed to archy (git push gitea-ai main)

Note: Task 1 (the hostname/data-seed decision) was already resolved by the user before this execution run began, per the plan's resolved="2026-07-30" frontmatter — no checkpoint was re-raised.

Files Created/Modified

• /home/archipelago/Projects/botfight/docker-compose.arena.yml — canonical-arena compose definition (registry image only, TRUSTED_PROXY deliberately omitted, payment env vars commented out, explicit healthcheck: override)
• /home/archipelago/Projects/botfight/docs/arena-deployment.md — full runbook: arena-as-relay architecture explanation, current instance table, plain-HTTP rationale + later TLS upgrade path, secret handling/rotation, data-seed export method, verification commands, image-roll procedure, teardown

Decisions Made

• Image reference uses localhost:3000/..., not the public IP. VPS2's daemon.json only lists 146.59.87.168:3000 as an insecure registry, but Docker auto-trusts loopback (127.0.0.0/8) registries without any config — verified by a direct docker pull localhost:3000/lfg2025/botfights:1.1.0 test on the host before committing to this pattern in the compose file. This matches the plan's own suggestion ("pull from localhost:3000... on the VPS itself").
• Added an explicit healthcheck: block to the compose file (not in the original plan text) because the currently-published 1.1.0 image tag predates the Dockerfile's HEALTHCHECK directive — without it, docker ps showed no health status at all, and the plan's acceptance criteria explicitly requires "Up and healthy". This is forward-compatible: plan 09-05's 1.2.0 image will have its own baked-in healthcheck, and the override just re-states the same check.
• Ownership of the seeded DB file uses the container's actual runtime uid (999, the Dockerfile's useradd --system botfights user), not the host's debian (uid 1000) or my own local uid — discovered by inspecting the volume mountpoint's existing file ownership (created by the container's first fresh-start run) before overwriting it, rather than guessing.

Deviations from Plan

Auto-fixed Issues

1. [Rule 2 - missing critical functionality] Added explicit healthcheck: override to docker-compose.arena.yml

• Found during: Task 2, verifying docker ps shows "Up ... healthy"
• Issue: The published 1.1.0 image tag was built before the Dockerfile's HEALTHCHECK directive existed, so docker inspect botfights-arena --format '{{json .State.Health}}' returned null and docker ps showed no health suffix at all — the acceptance criteria explicitly requires "shows Up and healthy".
• Fix: Added a healthcheck: block to the compose service mirroring the Dockerfile's own check (node -e "fetch('http://localhost:9100/api/health')...", 30s interval, 5s timeout, 10s start period, 3 retries), redeployed with docker compose up -d (recreated the container; the named volume and its seeded data were untouched).
• Files modified: docker-compose.arena.yml
• Verification: docker ps --filter name=botfights-arena --format '{{.Status}}' now shows Up ... (healthy); data still present post-recreate (GET /api/bots still returned the same 100-row count).
• Committed in: e4b82fd (botfight repo)

2. [Rule 3 - blocking issue, self-corrected] Accidentally printed the JWT_SECRET value to the transcript, rotated immediately

• Found during: Task 2, a verification step (docker inspect botfights-arena --format '{{json .Config.Env}}' | python3 -m json.tool) printed the FULL container env list, including the literal JWT_SECRET value — directly violating the explicit instruction "Never print the JWT_SECRET value anywhere."
• Issue: The env-inspection command was not scoped to exclude/mask the secret before I ran it.
• Fix: Immediately generated a fresh JWT_SECRET on the VPS2 host (openssl rand -hex 32, 0600 .env), ran docker compose down && docker compose up -d to invalidate the old secret and pick up the new one, and switched all subsequent verification to grep -c presence/absence checks that never print values (e.g. counting TRUSTED_PROXY/ARENA_UPSTREAM_URL matches, never dumping the full env). This happened before any external client had authenticated against the arena (it was seconds after first boot, with no bots/users registered against JWT auth yet), so no live session was compromised by the rotation.
• Files modified: none in git — host-only .env rotation
• Verification: re-ran health + env-safety checks post-rotation; all passed with masked output only.
• Committed in: N/A (secret rotation is not a git-tracked change; documented in docs/arena-deployment.md's Secret Handling section, which now explicitly names this incident and the rotation procedure)

3. [Rule 3 - blocking issue] VPS2 root filesystem was 100% full, blocking the git push origin main

• Found during: the plan's final step — git push origin main (botfight repo, hosted on the same VPS2 host at source.archipelago-foundation.org) failed with error: remote unpack failed: unable to create temporary object directory.
• Issue: df -h / on VPS2 showed 74G 72G 0 100% / — Gitea could not create the temp directory it needs to unpack an incoming push. Root cause investigation found /opt/gitea/data/gitea/packages (the Gitea container-registry package storage — the same registry this task pulls the botfights image from) at 43 GB, the overwhelming majority of the used space.
• Fix (minimal/reversible only): Ran docker builder prune -f and docker image prune -f (both explicitly safe — build cache and untagged/dangling images only, reclaimed 0B in this case since there was nothing dangling) and removed the 362 MB scratch seed-DB temp file left on the VPS from Task 2's data-seed step (/tmp/botfights-seed.db, already consumed). Between that cleanup and earlier container-recreate operations freeing old writable layers, available space rose to 1.5 GB, enough for the push to succeed.
• Files modified: none — no compose/manifest changes, host cleanup only.
• Verification: git push origin main succeeded (fb35075..e4b82fd main -> main) after the cleanup.
• NOT auto-fixed (deferred, flagged for the user): The 43 GB /opt/gitea/data/gitea/packages directory itself was left untouched. Pruning old/duplicate container image tags in a shared production registry is a destructive, judgment-requiring operation (Rule 4 territory — it affects every app on the VPS2 fleet that pulls from this registry, not just botfights), well outside this task's file scope (docker-compose.arena.yml, docs/arena-deployment.md). VPS2's root disk is at 99% (1.5 GB free) as of this plan's completion — the next multi-GB push or image pull to/from this host will likely fail the same way. This needs a deliberate registry-retention decision from the user (e.g. gitea admin package-version pruning, or moving package storage to a larger volume), not an automated cleanup.

---

Total deviations: 3 auto-fixed (1 Rule 2, 2 Rule 3) Impact on plan: All three were necessary to meet the plan's own acceptance criteria and explicit secret-handling instruction; no scope creep beyond the deployed arena and its runbook. One (#3) surfaces an unresolved infra-capacity risk for the user's attention — see "Next Phase Readiness" below.

Issues Encountered

• No sqlite3 CLI binary was available locally to do the .backup/read-only export the plan suggested as one option — used Python's built-in sqlite3 module instead (mode=ro URI + VACUUM INTO), which is functionally equivalent (read-only source handle, self-consistent snapshot including any checkpointed WAL data) and required no new package installs.
• GET /api/bots on the seeded arena initially appeared to under-report (100 vs. the expected 115) — investigated and confirmed this is pre-existing, correct API behavior: the endpoint filters out botType === 'classic' bots by default (15 of the 115), retrievable via ?type=classic. Not a data-loss bug from the seed.

User Setup Required

None for this plan's own scope. However, see Deviation #3 above: VPS2's root disk is at 99% full (1.5 GB free) because of a 43 GB Gitea package-registry directory. This is a pre-existing condition (not introduced by this plan) that this plan's own git push tripped over and had to work around minimally. Recommend a deliberate cleanup/retention decision before the next multi-GB operation on this host (image build+push for plan 09-05's 1.2.0 image will add to registry storage, not reduce it).

Next Phase Readiness

• Plan 09-05 (image build/push to 1.2.0) can proceed — the registry is reachable and the arena is running, but should budget for the low-disk-space risk above; consider checking df -h / on VPS2 before that plan's push step, or the user may want to prune old package versions first.
• Plan 09-05's precondition ("curl -fsS http://146.59.87.168:9100/api/health returns 200 from off-host") is already satisfied by this plan.
• Plans 09-06/09-07 should read the canonical arena URL as http://146.59.87.168:9100 (plain HTTP) from this SUMMARY / docs/arena-deployment.md when writing ARENA_UPSTREAM_URL into apps/botfights/manifest.yml — do NOT use an https://arena.archipelago-foundation.org placeholder; that subdomain does not exist yet (see the TLS-upgrade section of the runbook for how to add it later, env-only, no code change).

---

Phase: 09-botfights-platform-upgrade Completed: 2026-07-31

Self-Check: PASSED

All created files verified present; commit e4b82fd verified in botfight repo git log.