botfights/docs/arena-deployment.md

# Canonical Arena Deployment (VPS2)

This is the runbook for the one canonical, public BotFights arena. Every
node's local instance can proxy match/fighter state to a shared arena via
`ARENA_UPSTREAM_URL` — this document covers deploying the well-known default
one, on the Foundation's VPS2 host.

Nothing here is git-tracked automatically: nginx-proxy-manager's routing
config and the host `.env` (secrets) live only on the VPS2 host. This file is
the only record of how to reproduce or roll back the deployment.

## Architecture: arena-as-relay (read this first)

BotFights' shared-arena design is intentionally decentralized, not
hardcoded to one server:

- **Any node can host a public arena.** It's the exact same container image
  any node already runs — a "public arena" is just a BotFights instance with
  `ARENA_UPSTREAM_URL` **unset** (standalone mode) that other nodes point at.
- **Each node picks its own community** by setting `ARENA_UPSTREAM_URL` in
  its own manifest/environment. Unset = fully standalone, own local SQLite DB.
- **This VPS2 deployment is only the well-known default rendezvous** — like
  the vps2 FIPS anchor — not an authority baked into the code. Nothing in
  `botfight`'s server or frontend code hardcodes `146.59.87.168`; it is
  entirely an environment-variable choice made by whoever configures a node.
- The game UI is always served locally by each node; only match/fighter
  state lives wherever `ARENA_UPSTREAM_URL` points.
- **How to host your own arena:** deploy this exact `docker-compose.arena.yml`
  pattern (or even the node's normal `docker-compose.yml`) anywhere reachable,
  leave `ARENA_UPSTREAM_URL` unset on it, generate your own `JWT_SECRET`, and
  point whichever nodes you want in your community at
  `ARENA_UPSTREAM_URL=http://<your-host>:<port>`. There is no registration or
  allowlist step — the protocol is "point at a URL that speaks the BotFights
  API."

## Current canonical instance

| Field | Value |
|---|---|
| Host | VPS2, `debian@146.59.87.168` (docker, not podman — this is host infra, not an Archipelago node) |
| Directory | `/opt/botfights-arena/` |
| Compose file | `/opt/botfights-arena/docker-compose.yml` (copied from this repo's `docker-compose.arena.yml`, not symlinked — re-copy after edits) |
| Container name | `botfights-arena` |
| Image | `localhost:3000/lfg2025/botfights:1.1.0` (Gitea registry on the same host; `localhost:3000` resolves without any insecure-registry config because Docker trusts loopback registries by default — this is why the compose file uses `localhost:3000`, not the public `146.59.87.168:3000`, as the image ref) |
| Port | **9100** (verified free before binding; now bound — see `ss -tlnp` output in this phase's execution log) |
| Data volume | named volume `botfights-arena-data` → `/app/server/data` inside the container (host mountpoint: `docker volume inspect botfights-arena_botfights-arena-data --format '{{.Mountpoint}}'`) |
| **Canonical URL** | **`https://botfights.archipelago-foundation.org`** — TLS via nginx-proxy-manager + Let's Encrypt (user created DNS + proxy host 2026-07-30). Raw fallback: `http://146.59.87.168:9100` |

### Why plain HTTP on the raw port (no DNS/TLS this phase)

The user explicitly chose to skip creating a subdomain (e.g.
`arena.archipelago-foundation.org`), an nginx-proxy-manager proxy host, and a
Let's Encrypt certificate for this phase. Rationale:

- The node → arena hop is **server-side** (each node's Hono server proxies
  `/api/*` to `ARENA_UPSTREAM_URL`), never a browser fetch — so there is no
  mixed-content restriction that would otherwise force HTTPS.
- Cloud bots (server-to-server `curl`/HTTP clients) don't enforce
  browser-style mixed-content or certificate-pinning either.
- This keeps the deploy on the fast path for the 2026-07-31 demo — no DNS
  propagation wait, no cert-issuance step.

**Threat register note (T-09-16, accepted):** credentials (JWT bearer
tokens, NIP-98 auth headers, bot secrets) travel in plaintext over
`http://146.59.87.168:9100`. This is an accepted, recorded tradeoff, not an
oversight.

### Later TLS upgrade path (env-only, no code change)

When DNS/TLS is wanted:

1. Add an A record, e.g. `arena.archipelago-foundation.org` → `146.59.87.168`
   (GoDaddy `ns29/ns30.domaincontrol.com`, no wildcard — this needs its own
   record).
2. In nginx-proxy-manager (`https://146.59.87.168:81`, admin `lfg2025@proton.me`),
   add a new **Proxy Host**:
   - Domain: `arena.archipelago-foundation.org`
   - Scheme: `http`
   - Forward Hostname/IP: `146.59.87.168`
   - Forward Port: `9100`
   - Block Common Exploits: on
   - Websockets Support: on (`allow_websocket_upgrade=1` — required for any
     future websocket use; the current SSE fight-stream is plain HTTP
     chunked streaming and doesn't strictly need this, but it's the
     established pattern for every other subdomain on this host)
   - SSL tab: request a new Let's Encrypt certificate, force SSL
     (`ssl_forced=1`) — mirrors the existing `demo.`/`source.`/`fips.` hosts.
3. Change **only** the value every node reads: `ARENA_UPSTREAM_URL` in
   `apps/botfights/manifest.yml` (archy repo) from
   `http://146.59.87.168:9100` to `https://botfights.archipelago-foundation.org` — DONE 2026-07-30: the user created the DNS A record and the NPM proxy host with a Let's Encrypt cert; `TRUSTED_PROXY=1` was enabled on the arena at the same time (it now sits behind NPM).
   No code change — the reverse-proxy middleware and NIP-98 verification are
   both already origin-independent (path-only URL comparison).
4. Optionally keep `:9100` open as a fallback/legacy path, or firewall it
   down to only `127.0.0.1` once NPM is fronting it (`ports: - "127.0.0.1:9100:9100"`
   in the compose file) so the raw port is no longer publicly reachable.

## Secret handling

`JWT_SECRET` is generated **on the VPS2 host**, never in this repo, never in
a compose file value, never printed to a log or transcript:

```bash
# On VPS2, inside /opt/botfights-arena/:
umask 077
echo "JWT_SECRET=$(openssl rand -hex 32)" > .env
chmod 600 .env
```

`docker-compose.arena.yml` only ever references `${JWT_SECRET}` — the literal
value lives solely in `/opt/botfights-arena/.env` (mode `0600`, owned by
`debian`, outside any git repo).

**Rotation:** overwrite `.env` with a freshly-generated value, then
`docker compose down && docker compose up -d` (all existing sessions/JWTs
become invalid — bot `secret`/`bot_id` pairs used for `POST /api/bots` auth
are unaffected, only nostr-signer-issued JWTs expire).

**If a secret value is ever accidentally exposed** (e.g. printed by a
`docker inspect` command run without redaction): rotate immediately using
the steps above. This happened once during this phase's initial deployment
(caught and corrected the same session — the secret was rotated and the
container restarted before any external use).

## Data seed: full database copy (user decision 2026-07-30)

The arena was seeded from archi-dev-box's existing BotFights instance
(`/var/lib/archipelago/botfights/botfights.db`, 351 MB at the time of
export — 115 bots, 102,440 fights, `payments`/`bets` tables present but
empty).

**Export method (read-only, source never written to):**

```python
# Read-only URI connection — SQLite refuses writes on this handle.
# VACUUM INTO produces a compacted, self-consistent snapshot including
# any WAL-mode uncommitted-but-checkpointed data, without requiring write
# access to the source's -wal/-shm files.
import sqlite3
con = sqlite3.connect(
    'file:/var/lib/archipelago/botfights/botfights.db?mode=ro', uri=True)
con.execute("VACUUM INTO '/path/to/botfights-export.db'")
con.close()
```

Source file `mtime`/size were compared before and after the export and
confirmed byte-identical (`1782916151`, `367144960` bytes) — the export did
not touch the live node's database.

**Deploy steps used:**

1. `docker compose stop` on the arena (avoid the app writing to the volume
   mid-copy).
2. `scp` the exported `.db` file to VPS2, then as root:
   `cp` it into the named volume's host mountpoint as `botfights.db`,
   removing any stray `-wal`/`-shm` files from the fresh-start container run.
3. `chown` the file to uid/gid `999` — the container's non-root `botfights`
   system user (verify with `docker inspect botfights-arena --format
   '{{.Config.User}}'` and the uid `useradd --system` assigned it, since
   docker on this host does not use userns-remap — the host uid IS the
   container uid).
4. `docker compose start`.

**Result:** `GET /api/bots` returns **100** rows by default (the endpoint
filters out `botType === 'classic'` bots) — the remaining **15** classic-type
bots are visible via `GET /api/bots?type=classic`. `100 + 15 = 115`, matching
the source exactly. No data was lost; this is existing, unmodified API
filtering behavior, not an artifact of the copy.

## Verification (rerun any time to confirm the arena is healthy)

```bash
# On-host:
ssh debian@146.59.87.168 'curl -fsS http://127.0.0.1:9100/api/health'
# → {"status":"ok","name":"botfights"}

# Off-host (from archi-dev-box or any client with a path to VPS2):
curl -fsS --max-time 10 http://146.59.87.168:9100/api/health
curl -fsS --max-time 10 http://146.59.87.168:9100/api/bots   # expect 100 (+15 classic)
```

## Building and pushing `botfights:1.2.0` (plan 09-05)

`1.2.0` is the first image built after the arena-proxy middleware (09-01),
the nostr-only `GET /api/auth/me` auth fix (09-02), and the unified
`GET /api/docs/prompt` AI setup prompt (09-03) all landed on `main`. Build
from a clean checkout of `origin/main`:

```bash
cd /home/archipelago/Projects/botfight
git pull --ff-only origin main
# confirm all three wave-1 plans are present before building:
test -f server/src/middleware/arena-proxy.ts
grep -q "get('/me'" server/src/routes/auth.ts
grep -q "get('/prompt'" server/src/routes/docs.ts

podman build --build-arg CACHE_BUST=$(date +%s) \
  -t 146.59.87.168:3000/lfg2025/botfights:1.2.0 .

# Smoke test locally BEFORE pushing (spare port, no upstream configured):
podman run --rm -d --name botfights-smoketest -p 9199:9100 \
  -e NODE_ENV=production -e JWT_SECRET=$(openssl rand -hex 32) \
  146.59.87.168:3000/lfg2025/botfights:1.2.0
curl -fsS http://127.0.0.1:9199/api/health
curl -fsSi http://127.0.0.1:9199/api/docs/prompt | head -3   # expect 200, text/markdown
curl -si http://127.0.0.1:9199/api/auth/me | head -3         # expect 401, no Authorization header
podman rm -f botfights-smoketest

# Push (registry is plain HTTP; 146.59.87.168:3000 is already configured as an
# insecure registry in /etc/containers/registries.conf.d/archipelago.conf on
# this host, but --tls-verify=false is passed explicitly too):
podman login 146.59.87.168:3000 -u lfg2025 -p <token from Gitea admin, see infra memory note>
podman push --tls-verify=false 146.59.87.168:3000/lfg2025/botfights:1.2.0

# Verify from the registry side:
skopeo inspect --tls-verify=false docker://146.59.87.168:3000/lfg2025/botfights:1.2.0
```

**Build gotcha hit this session (pre-existing, unrelated to phase 09's own
code — fixed as an in-scope blocking-issue deviation):** `pnpm install
--frozen-lockfile` inside the `deps` build stage failed with
`ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. Root cause: an earlier commit
(`bcb323e`, March 2026) moved dependency `overrides` from `package.json`'s
`pnpm.overrides` key (a location modern pnpm no longer reads at all — see
its own deprecation warning) to `pnpm-workspace.yaml`'s `overrides:` key,
but only migrated 2 of 3 override entries and never regenerated
`pnpm-lock.yaml` to match. The Dockerfile's `corepack prepare pnpm@latest`
pulls whatever pnpm is current at build time, which enforces the
lockfile-vs-config check strictly. Fixed by: removing the dead `pnpm`
field from `package.json`, adding the missing `tar: '>=7.5.11'` override to
`pnpm-workspace.yaml` (alongside the two already there), and regenerating
`pnpm-lock.yaml` with `pnpm install --no-frozen-lockfile` — the resulting
lockfile diff contains **zero** `specifier:` changes (verified by grep),
only peer-dependency resolution-graph annotations from the newer pnpm
version explicitly listing `supports-color` as a peer. `pnpm install
--frozen-lockfile` and `tsc --noEmit -p server/tsconfig.json` both pass
clean against the regenerated lockfile.

**Result (this session, 2026-07-31):**

| Field | Value |
|---|---|
| Tag | `146.59.87.168:3000/lfg2025/botfights:1.2.0` |
| Digest | `sha256:854ea299...26e144` (short form; full digest recorded in `.planning/phases/09-botfights-platform-upgrade/09-05-SUMMARY.md` — re-derive any time with `skopeo inspect` above) |
| Built from | `botfight` repo `main` @ the commit carrying the `GET /api/fights/poll` route-order fix below (`2a343ac` + fix commit) |
| Local smoke test | `/api/health` → `{"status":"ok",...}`; `/api/docs/prompt` → 200 `text/markdown`; `/api/auth/me` (no auth) → 401; `/api/fights/poll` (registered bot) → 200 `{"pending":false}` |

**Deviation fixed in the same build pass:** `GET /api/fights/poll` (the
polling protocol BOT-02's unified prompt documents) was pre-existing-broken
— a `GET /:id` dynamic route registered earlier in `server/src/routes/fights.ts`
shadowed the later-registered static `GET /poll` route, so any polling bot's
poll request was matched as a fight-id lookup for id `"poll"` and always
returned `404 {"error":"Fight not found."}`. Reproduced independently on a
throwaway container with a fresh DB (not an artifact of the arena's seeded
data) before fixing. Fixed by moving the `/poll` and `/poll/respond` route
registrations above `/:id` in the router. This was necessary to meet this
plan's own acceptance criterion (bot auth via `GET /api/fights/poll` against
the public arena) and to make BOT-02's unified prompt's polling-mode
documentation actually true.

## Rolling the image tag

The tag is kept in exactly one place — `docker-compose.arena.yml`'s
`image:` line. To roll (e.g. plan 09-05's 1.2.0 build):

```bash
# 1. Edit docker-compose.arena.yml: image: localhost:3000/lfg2025/botfights:1.2.0
# 2. Copy to the host and redeploy:
scp docker-compose.arena.yml debian@146.59.87.168:/opt/botfights-arena/docker-compose.yml
ssh debian@146.59.87.168 'cd /opt/botfights-arena && docker compose pull && docker compose up -d'
```

The named volume (and therefore all arena data) is untouched by an image
roll — only `docker compose down -v` (never run this without intent) removes
it.

## Tearing it down

```bash
ssh debian@146.59.87.168 '
  cd /opt/botfights-arena
  docker compose down          # stops + removes the container; volume persists
  # docker compose down -v     # ALSO deletes the botfights-arena-data volume — destructive, confirm first
  # rm -rf /opt/botfights-arena  # only after confirming the volume is gone/backed up
'
```

## Ports already bound on VPS2 (verified 2026-07-30, re-check with `sudo ss -tlnp` before reusing)

22, 80, 81, 443, 2100, 2101, 2222, 3000, 3009, 5355, 7788, 8000, 8092, 8123,
8443, 8444, 9443, and now **9100** (this deployment).

## 1.2.0 public-contract verification (plan 09-05, 2026-07-31)

All checks below ran against `https://botfights.archipelago-foundation.org`
(never `127.0.0.1`/the raw port) after `docker compose pull && up -d` recreated
the container on the `botfights:1.2.0` tag (post-poll-fix build):

| Check | Result |
|---|---|
| `GET /api/health` | `{"status":"ok","name":"botfights"}` |
| `GET /api/docs/prompt` | 200, `text/markdown`, arena URL substituted 9×, zero leftover `{{ARENA_URL}}` tokens |
| `GET /api/auth/me` (no token) | 401 |
| `POST /api/bots` (anonymous, from off-host) | 200, id+secret issued; bot immediately visible in `GET /api/bots` |
| `GET /api/fights/poll` (bot auth via `Authorization: Bot id:secret`) | 200 `{"pending":false}` — see the `GET /:id` route-order fix above; this was 404 before it |
| `POST /api/queue/join/<botId>` → poll again | matched into a real fight within seconds; poll returned the live challenge payload |
| `GET /api/fights/<id>/stream` (SSE) | Incremental delivery confirmed: `spectator_count`/`ping` events at connection open, a second `ping` ~15s later, then `round_end`/`round_start`/`poll_challenge` in a fresh cluster ~4-5s after that — spread over a live 25s capture window, not buffered until stream close |
| `JWT_SECRET` survived the roll | `/opt/botfights-arena/.env` mtime predates this session's image rolls (unchanged); container's `JWT_SECRET` env still sources `${JWT_SECRET}` from that same file, not a freshly generated value |
| Data integrity | `GET /api/bots` → 101 (100 original + 1 test bot from an earlier verification pass), `?type=classic` → 15, unchanged/grown from the pre-roll 100+15 |

**Test bots left in the arena, clearly named per this plan's own naming
convention (no bot-deletion API exists in this codebase to remove them
cleanly):** `wavetest2`, `wavetest3` — both anonymous, harmless, real
fighters; consistent with the arena's existing `FIGHT_LOOP_ENABLED=true`
mock-bot background activity. `wavetest3` fought one live match as part of
verifying the SSE stream above.

## Verified cross-instance behaviour (plan 09-05 Task 3, 2026-07-31)

A throwaway `botfights:1.2.0` container (`botfights-proxytest`, port 9101,
no volume mount — nothing worth reading locally) ran on archi-dev-box with
`ARENA_UPSTREAM_URL=https://botfights.archipelago-foundation.org`, alongside
(never touching) the installed `botfights` app on port 9100 (image 1.1.0).

| Check | Result |
|---|---|
| Proxy instance has no local data of its own | Startup still seeds a local 100+15 mock-bot DB (unrelated background code path that runs regardless of `ARENA_UPSTREAM_URL`) — but every `/api/*` request is intercepted by `arena-proxy` before it ever reaches a local route handler, so that local data is never exposed through the API |
| `GET /api/bots` via the proxy instance | Returned 103 bots, including `wavetest2` and `wavetest3` — both registered directly against the arena in Task 2, never touching this instance. **This is the D1 proof: a fighter registered on one host is visible through a different instance that never stored it.** |
| Reverse direction: register via the proxy instance | `POST http://127.0.0.1:9101/api/bots {"name":"wavetest4"}` succeeded, and `wavetest4` was immediately visible in `GET https://botfights.archipelago-foundation.org/api/bots` directly |
| SSE through the proxy instance | `wavetest3` matched into a real fight; `GET http://127.0.0.1:9101/api/fights/<id>/stream` delivered `spectator_count`/`ping` at connection open and a second `ping` ~15s later — incremental, not buffered |
| `/api/health` bypass during a deliberate arena outage | `docker compose stop` on the VPS2 arena (seconds); `GET http://127.0.0.1:9101/api/health` still returned `200 {"status":"ok",...}` throughout — confirmed answered locally per `arena-proxy.ts`'s `LOCAL_BYPASS_PATHS`, never forwarded |
| `/api/bots` during the same outage, via the **canonical HTTPS URL** (fronted by nginx-proxy-manager since 2026-07-30) | `502`, but the body was NPM's own HTML error page, not the app's JSON — because NPM itself answers with a gateway-level 502 before the request ever reaches the stopped container; `fetch()` inside `arena-proxy.ts` succeeds against NPM and passes its response through verbatim. This supersedes the plan's original acceptance wording (written when the arena was still plain-HTTP/no-NPM); NPM 502ing here is expected, correct behavior for a proxy in front of a stopped upstream. |
| `/api/bots` during a second, separate short outage, via the **raw fallback port** (`http://146.59.87.168:9100`, no NPM in front) | `502 {"error":"Arena unreachable."}` — `arena-proxy.ts`'s own JSON degradation path (already unit-tested in 09-01), confirmed live against a real stopped upstream with no intermediary |
| Recovery | `docker compose start` on VPS2 both times; arena `healthy` again within seconds; the proxy instance's own subsequent requests succeeded immediately, no restart needed on the node side |
| Installed app isolation | `podman ps --filter name=botfights` showed the installed `botfights` app (port 9100, image `:1.1.0`) with its original container id and uptime, unaffected throughout; no `botfights-proxytest*` container remains after cleanup |

**Test bots registered during this task, left in the arena (same rationale
as Task 2 — clearly named, no delete API exists):** `wavetest4`.