Go global: worldwide BTC Map sync, marker clustering, response compression
- Sync now pulls every BTC Map place worldwide (~29k live), not just Madeira — dropped BTCMAP_CENTER_LAT/LON/RADIUS_KM entirely. - Incremental via a stored watermark (settings.btcmap_synced_since): only fetches what changed since the last sync, not the whole dataset every time. Handles deletions too (BTC Map delistings correctly evict the local place + cascade its claim/links, not just go unclaimed). - Real bug caught and fixed before it shipped: tried BTC Map's own recommended updated_since+limit pagination first, but their timestamps mix millisecond and whole-second precision, and naive string comparison across that isn't chronologically safe — silently truncated a real sync to ~4,000 of ~42,000 records with no error. Measured the alternative (single unpaginated request) instead: 2-3s for the full dataset, simpler and actually correct. Full writeup in services/btcmap.ts, regression test for the specific bug in services/btcmap.test.ts. - @fastify/compress added — the places list is now tens of thousands of rows, gzip/br/zstd auto-negotiated. - Also fixed while touching dependencies: @fastify/static had a real high-severity path-traversal/auth-bypass advisory (GHSA-pr96-94w5-mx2h et al) affecting the version we were pinned to — bumped to the patched 10.1.2. - Frontend: leaflet.markercluster (raw per-marker rendering doesn't scale to tens of thousands of points), cyberpunk-themed cluster icons to match the existing HUD styling, batch marker insertion (addLayers, not a per-marker addLayer loop) since that's dramatically faster at this scale. Default map view is now world-scale, recentering on the player's location if geolocation is available. - 60 backend tests passing throughout (7 new for the sync rewrite).
This commit is contained in:
@@ -4,7 +4,10 @@ A location-based capture game in the spirit of *Ingress*, built as a nostr-nativ
|
||||
web app. Real Bitcoin-accepting businesses from [BTC Map](https://btcmap.org)
|
||||
stand in for Ingress's portals — players physically visit them to claim them
|
||||
for their team (orange or green), then link claimed places into triangular
|
||||
control fields. First scope: **Madeira, Portugal**.
|
||||
control fields. **Global** — every BTC Map place worldwide (~29k live, plus
|
||||
historical deletions tracked so delisted businesses correctly drop out).
|
||||
Started as a Madeira, Portugal-only pilot; see git history/`DESIGN.md` for
|
||||
that phase.
|
||||
|
||||
See [`DESIGN.md`](./DESIGN.md) for the full research/design writeup.
|
||||
|
||||
@@ -12,7 +15,9 @@ See [`DESIGN.md`](./DESIGN.md) for the full research/design writeup.
|
||||
|
||||
- `server/` — Fastify + better-sqlite3 + nostr-tools (TypeScript), same pattern
|
||||
as [podsteadr](http://146.59.87.168:3000/ssmithx/podsteadr).
|
||||
- `frontend/` — Vue 3 + Vite + Pinia + Tailwind + Leaflet.
|
||||
- `frontend/` — Vue 3 + Vite + Pinia + Tailwind + Leaflet (+
|
||||
`leaflet.markercluster` — required at global scale; tens of thousands of
|
||||
raw markers would choke the DOM otherwise).
|
||||
- Auth is NIP-98 (signed-event login → session cookie), same as podsteadr.
|
||||
`frontend/public/nostr-provider.js` is vendored so the app can also be
|
||||
launched identity-aware from inside an Archipelago dashboard, the same way
|
||||
@@ -33,8 +38,43 @@ npm run dev # http://localhost:5173, proxies /api to :8096
|
||||
```
|
||||
|
||||
On first run, the `places` table is empty — call `POST /api/sync` (or click
|
||||
"Sync from BTC Map" in the UI) to pull the current Madeira dataset from BTC
|
||||
Map's public API.
|
||||
"Sync from BTC Map" in the UI) to pull the current worldwide dataset from BTC
|
||||
Map's public API (~29k places, a few seconds, single request — see "Global
|
||||
sync" below for why it's one request rather than paginated).
|
||||
|
||||
## Global sync
|
||||
|
||||
`POST /api/sync` pulls **every** BTC Map place worldwide via the
|
||||
chronological `/v4/places` endpoint, incrementally: a watermark
|
||||
(`btcmap_synced_since` in the `settings` table) tracks the last sync's
|
||||
newest `updated_at`, and each subsequent call only fetches what changed
|
||||
since then — call it as often as you like (e.g. a cron hitting it hourly)
|
||||
without re-pulling the full dataset every time.
|
||||
|
||||
Deleted places (BTC Map returns these too, via `include_deleted=true`) are
|
||||
evicted locally — `ON DELETE CASCADE` on `claims`/`links` means a delisted
|
||||
business correctly loses its claim and any links through it, not just the
|
||||
place row.
|
||||
|
||||
**Why one unpaginated request, not BTC Map's own recommended
|
||||
`updated_since`+`limit` pagination loop:** tried that first. BTC Map's
|
||||
`updated_at` timestamps mix millisecond-precision and whole-second values,
|
||||
and naive string comparison across mixed precision isn't chronologically
|
||||
safe (`"...27Z"` sorts *after* `"...27.137Z"` lexicographically, even though
|
||||
`.137` happened later within that same second) — this silently truncated a
|
||||
real sync to the first ~4,000 of ~42,000 records with no error, since the
|
||||
pagination cursor got stuck comparing strings instead of real timestamps.
|
||||
Measured the alternative instead: omitting `limit` entirely just returns the
|
||||
whole dataset — currently ~42k records including historical deletions, ~9MB
|
||||
uncompressed, in 2-3 seconds. Simpler and correct beats "matches the docs'
|
||||
suggested pattern" here. `services/btcmap.ts` has the full writeup;
|
||||
`services/btcmap.test.ts` has a regression test for the specific
|
||||
mixed-precision comparison bug. Revisit with real pagination (numeric/parsed
|
||||
timestamp cursor, not string comparison) if the dataset ever grows enough to
|
||||
make a single request unwieldy.
|
||||
|
||||
Response payloads (`/api/places` is the big one, tens of thousands of rows)
|
||||
are compressed (`@fastify/compress`, gzip/brotli/zstd auto-negotiated).
|
||||
|
||||
## Game rules (v0)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user