Go global: worldwide BTC Map sync, marker clustering, response compression

- Sync now pulls every BTC Map place worldwide (~29k live), not just
  Madeira — dropped BTCMAP_CENTER_LAT/LON/RADIUS_KM entirely.
- Incremental via a stored watermark (settings.btcmap_synced_since):
  only fetches what changed since the last sync, not the whole dataset
  every time. Handles deletions too (BTC Map delistings correctly evict
  the local place + cascade its claim/links, not just go unclaimed).
- Real bug caught and fixed before it shipped: tried BTC Map's own
  recommended updated_since+limit pagination first, but their timestamps
  mix millisecond and whole-second precision, and naive string comparison
  across that isn't chronologically safe — silently truncated a real
  sync to ~4,000 of ~42,000 records with no error. Measured the
  alternative (single unpaginated request) instead: 2-3s for the full
  dataset, simpler and actually correct. Full writeup in
  services/btcmap.ts, regression test for the specific bug in
  services/btcmap.test.ts.
- @fastify/compress added — the places list is now tens of thousands of
  rows, gzip/br/zstd auto-negotiated.
- Also fixed while touching dependencies: @fastify/static had a real
  high-severity path-traversal/auth-bypass advisory (GHSA-pr96-94w5-mx2h
  et al) affecting the version we were pinned to — bumped to the patched
  10.1.2.
- Frontend: leaflet.markercluster (raw per-marker rendering doesn't scale
  to tens of thousands of points), cyberpunk-themed cluster icons to
  match the existing HUD styling, batch marker insertion (addLayers, not
  a per-marker addLayer loop) since that's dramatically faster at this
  scale. Default map view is now world-scale, recentering on the
  player's location if geolocation is available.
- 60 backend tests passing throughout (7 new for the sync rewrite).
This commit is contained in:
2026-08-05 17:32:54 +00:00
parent 4fef930029
commit bb03bfc365
14 changed files with 464 additions and 200 deletions
+44 -4
View File
@@ -4,7 +4,10 @@ A location-based capture game in the spirit of *Ingress*, built as a nostr-nativ
web app. Real Bitcoin-accepting businesses from [BTC Map](https://btcmap.org)
stand in for Ingress's portals — players physically visit them to claim them
for their team (orange or green), then link claimed places into triangular
control fields. First scope: **Madeira, Portugal**.
control fields. **Global** — every BTC Map place worldwide (~29k live, plus
historical deletions tracked so delisted businesses correctly drop out).
Started as a Madeira, Portugal-only pilot; see git history/`DESIGN.md` for
that phase.
See [`DESIGN.md`](./DESIGN.md) for the full research/design writeup.
@@ -12,7 +15,9 @@ See [`DESIGN.md`](./DESIGN.md) for the full research/design writeup.
- `server/` — Fastify + better-sqlite3 + nostr-tools (TypeScript), same pattern
as [podsteadr](http://146.59.87.168:3000/ssmithx/podsteadr).
- `frontend/` — Vue 3 + Vite + Pinia + Tailwind + Leaflet.
- `frontend/` — Vue 3 + Vite + Pinia + Tailwind + Leaflet (+
`leaflet.markercluster` — required at global scale; tens of thousands of
raw markers would choke the DOM otherwise).
- Auth is NIP-98 (signed-event login → session cookie), same as podsteadr.
`frontend/public/nostr-provider.js` is vendored so the app can also be
launched identity-aware from inside an Archipelago dashboard, the same way
@@ -33,8 +38,43 @@ npm run dev # http://localhost:5173, proxies /api to :8096
```
On first run, the `places` table is empty — call `POST /api/sync` (or click
"Sync from BTC Map" in the UI) to pull the current Madeira dataset from BTC
Map's public API.
"Sync from BTC Map" in the UI) to pull the current worldwide dataset from BTC
Map's public API (~29k places, a few seconds, single request — see "Global
sync" below for why it's one request rather than paginated).
## Global sync
`POST /api/sync` pulls **every** BTC Map place worldwide via the
chronological `/v4/places` endpoint, incrementally: a watermark
(`btcmap_synced_since` in the `settings` table) tracks the last sync's
newest `updated_at`, and each subsequent call only fetches what changed
since then — call it as often as you like (e.g. a cron hitting it hourly)
without re-pulling the full dataset every time.
Deleted places (BTC Map returns these too, via `include_deleted=true`) are
evicted locally — `ON DELETE CASCADE` on `claims`/`links` means a delisted
business correctly loses its claim and any links through it, not just the
place row.
**Why one unpaginated request, not BTC Map's own recommended
`updated_since`+`limit` pagination loop:** tried that first. BTC Map's
`updated_at` timestamps mix millisecond-precision and whole-second values,
and naive string comparison across mixed precision isn't chronologically
safe (`"...27Z"` sorts *after* `"...27.137Z"` lexicographically, even though
`.137` happened later within that same second) — this silently truncated a
real sync to the first ~4,000 of ~42,000 records with no error, since the
pagination cursor got stuck comparing strings instead of real timestamps.
Measured the alternative instead: omitting `limit` entirely just returns the
whole dataset — currently ~42k records including historical deletions, ~9MB
uncompressed, in 2-3 seconds. Simpler and correct beats "matches the docs'
suggested pattern" here. `services/btcmap.ts` has the full writeup;
`services/btcmap.test.ts` has a regression test for the specific
mixed-precision comparison bug. Revisit with real pagination (numeric/parsed
timestamp cursor, not string comparison) if the dataset ever grows enough to
make a single request unwieldy.
Response payloads (`/api/places` is the big one, tens of thousands of rows)
are compressed (`@fastify/compress`, gzip/brotli/zstd auto-negotiated).
## Game rules (v0)