botfights/docs/arena-deployment.md

Canonical Arena Deployment (VPS2)

This is the runbook for the one canonical, public BotFights arena. Every node's local instance can proxy match/fighter state to a shared arena via ARENA_UPSTREAM_URL — this document covers deploying the well-known default one, on the Foundation's VPS2 host.

Nothing here is git-tracked automatically: nginx-proxy-manager's routing config and the host .env (secrets) live only on the VPS2 host. This file is the only record of how to reproduce or roll back the deployment.

Architecture: arena-as-relay (read this first)

BotFights' shared-arena design is intentionally decentralized, not hardcoded to one server:

• Any node can host a public arena. It's the exact same container image any node already runs — a "public arena" is just a BotFights instance with ARENA_UPSTREAM_URL unset (standalone mode) that other nodes point at.
• Each node picks its own community by setting ARENA_UPSTREAM_URL in its own manifest/environment. Unset = fully standalone, own local SQLite DB.
• This VPS2 deployment is only the well-known default rendezvous — like the vps2 FIPS anchor — not an authority baked into the code. Nothing in botfight's server or frontend code hardcodes 146.59.87.168; it is entirely an environment-variable choice made by whoever configures a node.
• The game UI is always served locally by each node; only match/fighter state lives wherever ARENA_UPSTREAM_URL points.
• How to host your own arena: deploy this exact docker-compose.arena.yml pattern (or even the node's normal docker-compose.yml) anywhere reachable, leave ARENA_UPSTREAM_URL unset on it, generate your own JWT_SECRET, and point whichever nodes you want in your community at ARENA_UPSTREAM_URL=http://<your-host>:<port>. There is no registration or allowlist step — the protocol is "point at a URL that speaks the BotFights API."

Current canonical instance

Field	Value
Host	VPS2, debian@146.59.87.168 (docker, not podman — this is host infra, not an Archipelago node)
Directory	/opt/botfights-arena/
Compose file	/opt/botfights-arena/docker-compose.yml (copied from this repo's docker-compose.arena.yml, not symlinked — re-copy after edits)
Container name	botfights-arena
Image	localhost:3000/lfg2025/botfights:1.1.0 (Gitea registry on the same host; localhost:3000 resolves without any insecure-registry config because Docker trusts loopback registries by default — this is why the compose file uses localhost:3000, not the public 146.59.87.168:3000, as the image ref)
Port	9100 (verified free before binding; now bound — see ss -tlnp output in this phase's execution log)
Data volume	named volume botfights-arena-data → /app/server/data inside the container (host mountpoint: docker volume inspect botfights-arena_botfights-arena-data --format '{{.Mountpoint}}')
Canonical URL	https://botfights.archipelago-foundation.org — TLS via nginx-proxy-manager + Let's Encrypt (user created DNS + proxy host 2026-07-30). Raw fallback: http://146.59.87.168:9100

Why plain HTTP on the raw port (no DNS/TLS this phase)

The user explicitly chose to skip creating a subdomain (e.g. arena.archipelago-foundation.org), an nginx-proxy-manager proxy host, and a Let's Encrypt certificate for this phase. Rationale:

• The node → arena hop is server-side (each node's Hono server proxies /api/* to ARENA_UPSTREAM_URL), never a browser fetch — so there is no mixed-content restriction that would otherwise force HTTPS.
• Cloud bots (server-to-server curl/HTTP clients) don't enforce browser-style mixed-content or certificate-pinning either.
• This keeps the deploy on the fast path for the 2026-07-31 demo — no DNS propagation wait, no cert-issuance step.

Threat register note (T-09-16, accepted): credentials (JWT bearer tokens, NIP-98 auth headers, bot secrets) travel in plaintext over http://146.59.87.168:9100. This is an accepted, recorded tradeoff, not an oversight.

Later TLS upgrade path (env-only, no code change)

When DNS/TLS is wanted:

1. Add an A record, e.g. arena.archipelago-foundation.org → 146.59.87.168 (GoDaddy ns29/ns30.domaincontrol.com, no wildcard — this needs its own record).
2. In nginx-proxy-manager (https://146.59.87.168:81, admin lfg2025@proton.me), add a new Proxy Host:
   • Domain: arena.archipelago-foundation.org
   • Scheme: http
   • Forward Hostname/IP: 146.59.87.168
   • Forward Port: 9100
   • Block Common Exploits: on
   • Websockets Support: on (allow_websocket_upgrade=1 — required for any future websocket use; the current SSE fight-stream is plain HTTP chunked streaming and doesn't strictly need this, but it's the established pattern for every other subdomain on this host)
   • SSL tab: request a new Let's Encrypt certificate, force SSL (ssl_forced=1) — mirrors the existing demo./source./fips. hosts.
3. Change only the value every node reads: ARENA_UPSTREAM_URL in apps/botfights/manifest.yml (archy repo) from http://146.59.87.168:9100 to https://botfights.archipelago-foundation.org — DONE 2026-07-30: the user created the DNS A record and the NPM proxy host with a Let's Encrypt cert; TRUSTED_PROXY=1 was enabled on the arena at the same time (it now sits behind NPM). No code change — the reverse-proxy middleware and NIP-98 verification are both already origin-independent (path-only URL comparison).
4. Optionally keep :9100 open as a fallback/legacy path, or firewall it down to only 127.0.0.1 once NPM is fronting it (ports: - "127.0.0.1:9100:9100" in the compose file) so the raw port is no longer publicly reachable.

Secret handling

JWT_SECRET is generated on the VPS2 host, never in this repo, never in a compose file value, never printed to a log or transcript:

# On VPS2, inside /opt/botfights-arena/:
umask 077
echo "JWT_SECRET=$(openssl rand -hex 32)" > .env
chmod 600 .env

docker-compose.arena.yml only ever references ${JWT_SECRET} — the literal value lives solely in /opt/botfights-arena/.env (mode 0600, owned by debian, outside any git repo).

Rotation: overwrite .env with a freshly-generated value, then docker compose down && docker compose up -d (all existing sessions/JWTs become invalid — bot secret/bot_id pairs used for POST /api/bots auth are unaffected, only nostr-signer-issued JWTs expire).

If a secret value is ever accidentally exposed (e.g. printed by a docker inspect command run without redaction): rotate immediately using the steps above. This happened once during this phase's initial deployment (caught and corrected the same session — the secret was rotated and the container restarted before any external use).

Data seed: full database copy (user decision 2026-07-30)

The arena was seeded from archi-dev-box's existing BotFights instance (/var/lib/archipelago/botfights/botfights.db, 351 MB at the time of export — 115 bots, 102,440 fights, payments/bets tables present but empty).

Export method (read-only, source never written to):

# Read-only URI connection — SQLite refuses writes on this handle.
# VACUUM INTO produces a compacted, self-consistent snapshot including
# any WAL-mode uncommitted-but-checkpointed data, without requiring write
# access to the source's -wal/-shm files.
import sqlite3
con = sqlite3.connect(
    'file:/var/lib/archipelago/botfights/botfights.db?mode=ro', uri=True)
con.execute("VACUUM INTO '/path/to/botfights-export.db'")
con.close()

Source file mtime/size were compared before and after the export and confirmed byte-identical (1782916151, 367144960 bytes) — the export did not touch the live node's database.

Deploy steps used:

1. docker compose stop on the arena (avoid the app writing to the volume mid-copy).
2. scp the exported .db file to VPS2, then as root: cp it into the named volume's host mountpoint as botfights.db, removing any stray -wal/-shm files from the fresh-start container run.
3. chown the file to uid/gid 999 — the container's non-root botfights system user (verify with docker inspect botfights-arena --format '{{.Config.User}}' and the uid useradd --system assigned it, since docker on this host does not use userns-remap — the host uid IS the container uid).
4. docker compose start.

Result: GET /api/bots returns 100 rows by default (the endpoint filters out botType === 'classic' bots) — the remaining 15 classic-type bots are visible via GET /api/bots?type=classic. 100 + 15 = 115, matching the source exactly. No data was lost; this is existing, unmodified API filtering behavior, not an artifact of the copy.

Verification (rerun any time to confirm the arena is healthy)

# On-host:
ssh debian@146.59.87.168 'curl -fsS http://127.0.0.1:9100/api/health'
# → {"status":"ok","name":"botfights"}

# Off-host (from archi-dev-box or any client with a path to VPS2):
curl -fsS --max-time 10 http://146.59.87.168:9100/api/health
curl -fsS --max-time 10 http://146.59.87.168:9100/api/bots   # expect 100 (+15 classic)

Building and pushing botfights:1.2.0 (plan 09-05)

1.2.0 is the first image built after the arena-proxy middleware (09-01), the nostr-only GET /api/auth/me auth fix (09-02), and the unified GET /api/docs/prompt AI setup prompt (09-03) all landed on main. Build from a clean checkout of origin/main:

cd /home/archipelago/Projects/botfight
git pull --ff-only origin main
# confirm all three wave-1 plans are present before building:
test -f server/src/middleware/arena-proxy.ts
grep -q "get('/me'" server/src/routes/auth.ts
grep -q "get('/prompt'" server/src/routes/docs.ts

podman build --build-arg CACHE_BUST=$(date +%s) \
  -t 146.59.87.168:3000/lfg2025/botfights:1.2.0 .

# Smoke test locally BEFORE pushing (spare port, no upstream configured):
podman run --rm -d --name botfights-smoketest -p 9199:9100 \
  -e NODE_ENV=production -e JWT_SECRET=$(openssl rand -hex 32) \
  146.59.87.168:3000/lfg2025/botfights:1.2.0
curl -fsS http://127.0.0.1:9199/api/health
curl -fsSi http://127.0.0.1:9199/api/docs/prompt | head -3   # expect 200, text/markdown
curl -si http://127.0.0.1:9199/api/auth/me | head -3         # expect 401, no Authorization header
podman rm -f botfights-smoketest

# Push (registry is plain HTTP; 146.59.87.168:3000 is already configured as an
# insecure registry in /etc/containers/registries.conf.d/archipelago.conf on
# this host, but --tls-verify=false is passed explicitly too):
podman login 146.59.87.168:3000 -u lfg2025 -p <token from Gitea admin, see infra memory note>
podman push --tls-verify=false 146.59.87.168:3000/lfg2025/botfights:1.2.0

# Verify from the registry side:
skopeo inspect --tls-verify=false docker://146.59.87.168:3000/lfg2025/botfights:1.2.0

Build gotcha hit this session (pre-existing, unrelated to phase 09's own code — fixed as an in-scope blocking-issue deviation): pnpm install --frozen-lockfile inside the deps build stage failed with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH. Root cause: an earlier commit (bcb323e, March 2026) moved dependency overrides from package.json's pnpm.overrides key (a location modern pnpm no longer reads at all — see its own deprecation warning) to pnpm-workspace.yaml's overrides: key, but only migrated 2 of 3 override entries and never regenerated pnpm-lock.yaml to match. The Dockerfile's corepack prepare pnpm@latest pulls whatever pnpm is current at build time, which enforces the lockfile-vs-config check strictly. Fixed by: removing the dead pnpm field from package.json, adding the missing tar: '>=7.5.11' override to pnpm-workspace.yaml (alongside the two already there), and regenerating pnpm-lock.yaml with pnpm install --no-frozen-lockfile — the resulting lockfile diff contains zero specifier: changes (verified by grep), only peer-dependency resolution-graph annotations from the newer pnpm version explicitly listing supports-color as a peer. pnpm install --frozen-lockfile and tsc --noEmit -p server/tsconfig.json both pass clean against the regenerated lockfile.

Result (this session, 2026-07-31):

Field	Value
Tag	146.59.87.168:3000/lfg2025/botfights:1.2.0
Digest	sha256:854ea299...26e144 (short form; full digest recorded in .planning/phases/09-botfights-platform-upgrade/09-05-SUMMARY.md — re-derive any time with skopeo inspect above)
Built from	botfight repo main @ the commit carrying the GET /api/fights/poll route-order fix below (2a343ac + fix commit)
Local smoke test	/api/health → {"status":"ok",...}; /api/docs/prompt → 200 text/markdown; /api/auth/me (no auth) → 401; /api/fights/poll (registered bot) → 200 {"pending":false}

Deviation fixed in the same build pass: GET /api/fights/poll (the polling protocol BOT-02's unified prompt documents) was pre-existing-broken — a GET /:id dynamic route registered earlier in server/src/routes/fights.ts shadowed the later-registered static GET /poll route, so any polling bot's poll request was matched as a fight-id lookup for id "poll" and always returned 404 {"error":"Fight not found."}. Reproduced independently on a throwaway container with a fresh DB (not an artifact of the arena's seeded data) before fixing. Fixed by moving the /poll and /poll/respond route registrations above /:id in the router. This was necessary to meet this plan's own acceptance criterion (bot auth via GET /api/fights/poll against the public arena) and to make BOT-02's unified prompt's polling-mode documentation actually true.

Rolling the image tag

The tag is kept in exactly one place — docker-compose.arena.yml's image: line. To roll (e.g. plan 09-05's 1.2.0 build):

# 1. Edit docker-compose.arena.yml: image: localhost:3000/lfg2025/botfights:1.2.0
# 2. Copy to the host and redeploy:
scp docker-compose.arena.yml debian@146.59.87.168:/opt/botfights-arena/docker-compose.yml
ssh debian@146.59.87.168 'cd /opt/botfights-arena && docker compose pull && docker compose up -d'

The named volume (and therefore all arena data) is untouched by an image roll — only docker compose down -v (never run this without intent) removes it.

Tearing it down

ssh debian@146.59.87.168 '
  cd /opt/botfights-arena
  docker compose down          # stops + removes the container; volume persists
  # docker compose down -v     # ALSO deletes the botfights-arena-data volume — destructive, confirm first
  # rm -rf /opt/botfights-arena  # only after confirming the volume is gone/backed up
'

Ports already bound on VPS2 (verified 2026-07-30, re-check with sudo ss -tlnp before reusing)

22, 80, 81, 443, 2100, 2101, 2222, 3000, 3009, 5355, 7788, 8000, 8092, 8123, 8443, 8444, 9443, and now 9100 (this deployment).

1.2.0 public-contract verification (plan 09-05, 2026-07-31)

All checks below ran against https://botfights.archipelago-foundation.org (never 127.0.0.1/the raw port) after docker compose pull && up -d recreated the container on the botfights:1.2.0 tag (post-poll-fix build):

Check	Result
GET /api/health	{"status":"ok","name":"botfights"}
GET /api/docs/prompt	200, text/markdown, arena URL substituted 9×, zero leftover {{ARENA_URL}} tokens
GET /api/auth/me (no token)	401
POST /api/bots (anonymous, from off-host)	200, id+secret issued; bot immediately visible in GET /api/bots
GET /api/fights/poll (bot auth via Authorization: Bot id:secret)	200 {"pending":false} — see the GET /:id route-order fix above; this was 404 before it
POST /api/queue/join/<botId> → poll again	matched into a real fight within seconds; poll returned the live challenge payload
GET /api/fights/<id>/stream (SSE)	Incremental delivery confirmed: spectator_count/ping events at connection open, a second ping ~15s later, then round_end/round_start/poll_challenge in a fresh cluster ~4-5s after that — spread over a live 25s capture window, not buffered until stream close
JWT_SECRET survived the roll	/opt/botfights-arena/.env mtime predates this session's image rolls (unchanged); container's JWT_SECRET env still sources ${JWT_SECRET} from that same file, not a freshly generated value
Data integrity	GET /api/bots → 101 (100 original + 1 test bot from an earlier verification pass), ?type=classic → 15, unchanged/grown from the pre-roll 100+15

Test bots left in the arena, clearly named per this plan's own naming convention (no bot-deletion API exists in this codebase to remove them cleanly): wavetest2, wavetest3 — both anonymous, harmless, real fighters; consistent with the arena's existing FIGHT_LOOP_ENABLED=true mock-bot background activity. wavetest3 fought one live match as part of verifying the SSE stream above.

Verified cross-instance behaviour (plan 09-05 Task 3, 2026-07-31)

A throwaway botfights:1.2.0 container (botfights-proxytest, port 9101, no volume mount — nothing worth reading locally) ran on archi-dev-box with ARENA_UPSTREAM_URL=https://botfights.archipelago-foundation.org, alongside (never touching) the installed botfights app on port 9100 (image 1.1.0).

Check	Result
Proxy instance has no local data of its own	Startup still seeds a local 100+15 mock-bot DB (unrelated background code path that runs regardless of ARENA_UPSTREAM_URL) — but every /api/* request is intercepted by arena-proxy before it ever reaches a local route handler, so that local data is never exposed through the API
GET /api/bots via the proxy instance	Returned 103 bots, including wavetest2 and wavetest3 — both registered directly against the arena in Task 2, never touching this instance. This is the D1 proof: a fighter registered on one host is visible through a different instance that never stored it.
Reverse direction: register via the proxy instance	POST http://127.0.0.1:9101/api/bots {"name":"wavetest4"} succeeded, and wavetest4 was immediately visible in GET https://botfights.archipelago-foundation.org/api/bots directly
SSE through the proxy instance	wavetest3 matched into a real fight; GET http://127.0.0.1:9101/api/fights/<id>/stream delivered spectator_count/ping at connection open and a second ping ~15s later — incremental, not buffered
/api/health bypass during a deliberate arena outage	docker compose stop on the VPS2 arena (seconds); GET http://127.0.0.1:9101/api/health still returned 200 {"status":"ok",...} throughout — confirmed answered locally per arena-proxy.ts's LOCAL_BYPASS_PATHS, never forwarded
/api/bots during the same outage, via the canonical HTTPS URL (fronted by nginx-proxy-manager since 2026-07-30)	502, but the body was NPM's own HTML error page, not the app's JSON — because NPM itself answers with a gateway-level 502 before the request ever reaches the stopped container; fetch() inside arena-proxy.ts succeeds against NPM and passes its response through verbatim. This supersedes the plan's original acceptance wording (written when the arena was still plain-HTTP/no-NPM); NPM 502ing here is expected, correct behavior for a proxy in front of a stopped upstream.
/api/bots during a second, separate short outage, via the raw fallback port (http://146.59.87.168:9100, no NPM in front)	502 {"error":"Arena unreachable."} — arena-proxy.ts's own JSON degradation path (already unit-tested in 09-01), confirmed live against a real stopped upstream with no intermediary
Recovery	docker compose start on VPS2 both times; arena healthy again within seconds; the proxy instance's own subsequent requests succeeded immediately, no restart needed on the node side
Installed app isolation	podman ps --filter name=botfights showed the installed botfights app (port 9100, image :1.1.0) with its original container id and uptime, unaffected throughout; no botfights-proxytest* container remains after cleanup

Test bots registered during this task, left in the arena (same rationale as Task 2 — clearly named, no delete API exists): wavetest4.