docs/dynamic-mint-data.md

@@ -0,0 +1,361 @@
+# Dynamic mint data: analysis
+
+**Status:** analysis only. No code or config was changed.
+
+**Goal:** new and updated mints, stats, and reviews should appear on the list (and related list surfaces) immediately, without waiting for the nightly static rebuild.
+
+**Current architecture (important correction):** nginx does **not** serve `web/dist` as files. Production is:
+
+```
+browser → nginx :443 → cashumints-site (web/server.mjs :8789) → published copy of dist
+                      → cashumints_api  (Hono :8788)          → /api/*, /icons/*
+```
+
+`cashumints-web.timer` rebuilds at **03:30** daily, then `rsync`s `web/dist/` to `WEB_ROOT` (`/var/lib/cashumints/web`). Markup is a snapshot of whatever the API returned at that build.
+
+---
+
+## 1. Data flow audit
+
+There are **no Astro content collections**. Every dynamic page either fetches the Hono API at **build time** (`web/src/lib/api.ts`, `API_URL`, default `http://127.0.0.1:8787` / prod `8788`) or hydrates in the browser from `/api/…` and/or Nostr.
+
+Astro “islands” here are **not** React/Vue/Svelte with `client:load`. There is **zero** `client:*` usage. Runtime behaviour is vanilla `<script>` modules in `.astro` files, bundled by Vite, re-run on view transitions via `onReady()`.
+
+### 1.1 Every route under `web/src/pages`
+
+| Route | `getStaticPaths` | Build-time data | Client-side at runtime |
+|---|---|---|---|
+| `[...locale]/index.astro` (`/`, `/es`, …) | `localePaths` (one HTML page per locale) | `fetchMints`, `fetchFedimints`, `fetchLnurlMints`, `fetchStats`, `fetchHealth`; N+1 `fetchMint` / `fetchFedimint` / `fetchLnurlMint` for the top 6 of each; `fetchLatestReviews()` from relays via `nostr-build.ts` | **PulseTicker** refreshes `/api/stats` + `/api/health`. Search form is GET to `/mints`. Review carousel is **not** re-queried. ColorBends is WebGL only. |
+| `[...locale]/mints.astro` | `localePaths` | `fetchMints()` then **one `fetchMint(host)` per mint** for NUT chips | Search / sort / hide-offline only rearrange **already-rendered** cards. **No live refetch.** |
+| `[...locale]/fedimints.astro` | `localePaths` | `fetchFedimints()` (no N+1; no NUT chips) | Same client filter/sort as `/mints`. **No live refetch.** |
+| `[...locale]/lnurl-mints.astro` | `localePaths` | `fetchLnurlMints()` + N+1 `fetchLnurlMint` for chips | Same. **No live refetch.** |
+| `[...locale]/mint/[host].astro` | `localePathsFor` over **every cashu mint known at build** | `fetchMints()` + `fetchMint(host)` per mint, shared across locales | **MintLive** → `GET /api/mints/:host`. **Reviews** → Nostr via `nostr-tools` `SimplePool`. |
+| `[...locale]/fedimint/[id].astro` | same pattern, `fetchFedimints` | federation detail | Same two islands. |
+| `[...locale]/lnurl-mint/[host].astro` | same pattern, `fetchLnurlMints` | LNURL detail | Same two islands. |
+| `[...locale]/reviews.astro` | `localePaths` | `fetchAllListings()`, `fetchStats()`, `fetchReviewFeed(…, 30)` from relays | **Re-queries relays** (`feed-client.ts` `loadFeed`), paginates, filters. This is the working “live list” pattern. |
+| `[...locale]/wallets.astro` | `localePaths` | Hardcoded wallet array in the page | None |
+| `[...locale]/about.astro` | `localePaths` | `fetchStats()` for counts in copy | None |
+| `[...locale]/privacy.astro`, `terms.astro`, `disclaimer.astro` | `localePaths` | i18n catalogs only | None |
+| `[...locale]/404.astro` + `pages/404.astro` | prefixed locales / English `/404` | None (static 404 shell) | **NotFound resolver:** `GET /api/mints/:host`, on miss `POST /api/index`, then reviews panel |
+| `sitemap.xml.ts` | n/a (endpoint at build) | `fetchMints` / `fetchFedimints` / `fetchLnurlMints`; `isIndexableMint` filter | n/a |
+| `robots.txt.ts` | n/a | `SITE_URL` only; `Disallow: /api/`, `/icons/` | n/a |
+
+24 locales (`en` plus 23 prefixes). List pages are ~24 HTML files each. Mint detail pages are `mints × locales` (README ballpark: ~55 cashu mints × 24 ≈ 1,300+ pages, plus federations and LNURL).
+
+### 1.2 Why `/mints` shows only a subset, with stale stats and reviews
+
+**Not pagination.** `fetchMints()` calls `GET /api/mints?type=cashu` with **no `limit`**. `listMints()` in the API returns every matching row, sorted by score. The optional `?limit=` query is unused by the site.
+
+**Not an `isIndexableMint` filter on the grid.** Unreachable, unreviewed mints still render as cards. That predicate only drops them from JSON-LD `ItemList` and the sitemap.
+
+**The list is a frozen HTML snapshot.** `mints.astro` does:
+
+```ts
+const mints = await fetchMints();
+const capabilities = await Promise.all(
+  mints.map((mint) => fetchMint(mint.host).then(…).catch(() => null)),
+);
+```
+
+then maps every item to `<MintCard>`. After publish, those cards do not change until the next `cashumints-web.timer` run.
+
+What that freezes:
+
+| Card field | Source at build | Live counterpart |
+|---|---|---|
+| Presence of the mint | API `mints` table at 03:30 | Discovery + `POST /api/index` write new rows immediately |
+| `review_count`, `rating_avg`, `score`, `last_review_at` | SQL aggregates of **ingested** Nostr reviews | Relays (and the API, once discovery has ingested) |
+| `status`, `last_online` | last probe stored in the DB | `GET /api/mints/:host` (what MintLive already uses) |
+| Melt-only / frozen chip | N+1 detail fetch of cached `/v1/info` | same detail payload |
+
+**Why a mint can exist “on the site” but not on the list:** the 404 resolver (`NotFound.astro`) looks up `/mint/{host}` against the **live** API and, if missing, calls `POST /api/index`. That mint becomes reviewable at once. It does **not** get a card on `/mints` until rebuild. Same for Review-by-URL. The README states this as intended:
+
+> Mint pages are prerendered, so new mints and new review counts appear at the next build. … an unbuilt mint still resolves through the client-side fallback on the 404 page.
+
+**Why reviews look wrong on the list and right on the detail page:** `/mints` never talks to Nostr. It prints `MintListItem.review_count` / `rating_avg` from the API snapshot. The detail page’s `Reviews.astro` calls `loadReviews(subject)` against relays and replaces the panel. A review written today is on the detail page as soon as relays answer; the list card still shows yesterday’s count.
+
+**Home page extra subset:** `mints.slice(0, 6)` (and the same for federations / LNURL). “Latest reviews” is a **build-time** relay read (`fetchLatestReviews(…, 10)`), unlike `/reviews`, which re-queries on load.
+
+### 1.3 How the mint detail page fetches Nostr (the pattern that works)
+
+**Component:** `web/src/components/Reviews.astro`  
+Root: `[data-reviews-panel]` with `data-subject={JSON.stringify(subject)}`.  
+Script imports `loadReviews` / `loadProfiles` from `web/src/lib/reviews-client.ts`.
+
+**Library:** [`nostr-tools`](https://github.com/nbd-wtf/nostr-tools) **v2.10.4** — `SimplePool` from `nostr-tools/pool`, plus `nip19`. **Not NDK.**
+
+**Relays** (`DEFAULT_RELAYS` in `shared/src/nostr.ts`, overridable with `PUBLIC_REVIEW_RELAYS`):
+
+- `wss://relay.cashumints.space`
+- `wss://nos.lol`
+- `wss://relay.azzamo.net`
+- `wss://relay.snort.social`
+- `wss://relay.primal.net`
+
+Profiles add `wss://purplepag.es` and `wss://relay.nostr.net`.
+
+**Query:** kind `38000` (addressable reviews), two OR’d filters, `limit: 500`, `maxWait: 8000` ms:
+
+1. `#d` = mint pubkey (plus LNURL alternate ids) and `#k` = announcement kind
+2. `#u` = URL spellings (legacy Cashu reviews; federations skip this)
+
+Newest event per author wins. In-tab cache 5 minutes. Empty result is treated as a relay error if the API had counted reviews.
+
+**Live status (API, not Nostr):** `MintLive.astro` is a hidden `[data-mint-live={host}]` marker. Its script `fetch(`${apiBase}/api/mints/${host}`)` and patches the status chip, banner, limits, and NUT/module/feature rows. `apiBase` is `import.meta.env.PUBLIC_API_URL`, empty in production → same-origin `/api/…`.
+
+**Unbuilt mint URLs:** `server.mjs` serves the locale 404. `NotFound.astro` derives the address from the path, `GET /api/mints/:host`, then `POST /api/index` on miss, then dispatches `cashumints:subject` so the same Reviews island starts.
+
+---
+
+## 2. API coverage
+
+All routes are in `api/src/server.ts`. CORS is enabled on `/api/*` and `/icons/*` (`hono/cors`, default allow-all). Production uses **same origin** (`PUBLIC_API_URL=` empty; nginx proxies `/api/` and `/icons/`), so browsers never hit a cross-origin API unless that env is set.
+
+### 2.1 Existing endpoints
+
+#### `GET /api/mints`
+
+- Query: `?type=cashu|fedimint|lnurl` (unknown type → empty array); optional `?limit=N`
+- No type filter → **entire index**, each item tagged `type`
+- **No default limit.** Full list.
+- Response: `MintListItem[]`
+
+```ts
+{
+  url: string;
+  host: string;
+  name: string | null;
+  icon: string | null;          // "/icons/…" or null
+  type: 'cashu' | 'fedimint' | 'lnurl' | string;
+  status: 'online' | 'degraded' | 'offline' | 'unknown' | 'announced';
+  last_online: number | null;   // unix
+  review_count: number;         // ingested, one-per-author
+  rating_avg: number | null;    // 1 decimal
+  score: number;                // Bayesian, from those aggregates
+  last_review_at: number | null;
+  version: string | null;
+}
+```
+
+**This is enough to rebuild the `/mints` grid** (name, icon, rating, count, status, sort keys). It is **not** enough for melt-only / frozen chips (`nuts` / capabilities) or for true sentiment bars (`rating_distribution`). Those currently require N+1 `GET /api/mints/:host`.
+
+#### `GET /api/mints/:host`
+
+- 404 `{ error, message }` if unknown
+- Response: `MintDetail` = list item plus:
+
+```ts
+{
+  description, pubkey, info /* NUT-06 */, nuts: string[],
+  first_seen, last_probe, updated_at,
+  rating_distribution: { '1'..'5': number },
+  reviews_90d, uptime_30d, probes_recent: { ts, ok, latency_ms }[],
+  // fedimint extras when type=fedimint: federation_id, invite_codes, modules, …
+  // lnurl extras when type=lnurl: features, funding_available, withdraw bounds, …
+}
+```
+
+Used by MintLive and the 404 resolver. **No review bodies.**
+
+#### `GET /api/stats`
+
+In-process memo **60s**. Cashu `mints_*` counts are cashu-only.
+
+```ts
+{
+  mints_total, mints_online, mints_offline, mints_degraded,
+  reviews_total, last_review_at, updated_at,
+  cashu_total,  // same as mints_total
+  fedimint_total, fedimint_online, fedimint_offline, fedimint_announced, fedimint_reviews,
+  lnurl_total, lnurl_online, lnurl_offline, lnurl_degraded_funding, lnurl_reviews
+}
+```
+
+PulseTicker already consumes this live.
+
+#### `GET /api/health`
+
+`Cache-Control: no-store`. `{ status, uptime_s, last_probe_at, last_discovery_at, mints_tracked, updated_at }`. 503 when degraded.
+
+#### `POST /api/index`
+
+The only write. Body `{ type, input }`. Rate limit **10/hour/IP** (`INDEX_RATE_LIMIT`). Success 200/201: `MintDetail & { existing, indexed_from? }`. Failures: `bad_input`, `blocked_host`, `wrong_type`, `unverifiable`, `rate_limited`, etc.
+
+#### `GET /icons/:file`
+
+Static files, `Cache-Control: public, max-age=86400`.
+
+### 2.2 Gaps
+
+| Need | Exists? | Notes |
+|---|---|---|
+| Full mint / federation / LNURL lists | **Yes** | `/api/mints?type=…` |
+| Per-mint stats (status, uptime, distribution, probes) | **Yes** | `/api/mints/:host` |
+| Review **bodies** / recent review text | **No HTTP endpoint** | Live only via Nostr (`reviews-client.ts`, `feed-client.ts`). API stores ratings for aggregates, not content for the UI. |
+| List payload with NUT chips / LNURL chips | **No** | List item has no `nuts`, no `rating_distribution`. Today: N+1 at **build**. Client-side N+1 per visitor would be wasteful. |
+| Global review feed JSON | **No** | `/reviews` talks to relays, not the API. |
+| Snapshot JSON in `dist` | **None** | No checked-in mint dump. Stale data **is** the prerendered HTML. |
+
+Discovery itself is not the list cap (`QUERY_LIMIT` 500 × `MAX_PAGES` 20). The visible cap is **whatever was in the DB when `astro build` ran**.
+
+---
+
+## 3. Option analysis
+
+### 3a. Client-side hydration (keep static build)
+
+Keep `output: 'static'`, `server.mjs`, nginx, and the timer. On `/mints` (and twins), fetch the live API after load and rebuild the grid — the same idea as Reviews / PulseTicker / MintLive.
+
+**What already exists to copy**
+
+- PulseTicker: prerender numbers, then `fetch(`${apiBase}/api/stats`)`.
+- `/reviews`: prerender ~30 cards, then `loadFeed()` replaces them.
+- MintLive: patch in place from `/api/mints/:host`.
+- `/mints` script already sorts/filters on `data-*` (`data-score`, `data-rating`, `data-reviews`, `data-last-review`, `data-status`, …). New cards only need those attributes.
+
+**There are no `client:load` islands to add.** Work is extra `<script>` in the list pages (or a shared module) plus an HTML builder for a card, because `MintCard.astro` cannot run in the browser.
+
+Concrete changes:
+
+| File | Change |
+|---|---|
+| `web/src/pages/[...locale]/mints.astro` | After paint, `fetch('/api/mints?type=cashu')`, rebuild `[data-mint-grid]`, update `[data-result-count]`. Keep prerendered cards as noscript / first-paint fallback. |
+| `web/src/pages/[...locale]/fedimints.astro` | Same, `?type=fedimint`. |
+| `web/src/pages/[...locale]/lnurl-mints.astro` | Same, `?type=lnurl`. |
+| `web/src/lib/mint-cards.ts` (**new**) | Browser HTML for one card, mirroring `MintCard.astro` (same pattern as `review-cards.ts`). |
+| `web/src/lib/api.ts` | Optional: a tiny browser `fetch` helper using `apiBase` from `client.ts` (build-time `API_URL` must **not** ship to the browser). |
+| `web/src/pages/[...locale]/index.astro` | Optional but needed for “immediately”: refresh top-6 grids from the same list endpoint; optionally reuse `/reviews`’s `loadFeed` for the carousel (today that strip is build-only). |
+| `web/src/components/MintCard.astro` | Untouched if the JS builder is a parallel renderer; or extract shared markup helpers. |
+
+**Chips:** either (i) leave prerendered chips stale, (ii) add `nuts` / chip flags to `MintListItem` + `toListItem()` in `api/src/queries.ts` (small, one-time API change), or (iii) N+1 detail fetches from every browser (do not do this).
+
+**New mint click path:** hydrated list → `/mint/{newhost}` → 404 HTML → NotFound resolver → live page. Already designed. No SSR required for that URL to work.
+
+**Dependencies:** none new (`nostr-tools` already in `web`). No adapter. No nginx / systemd change for the site server.
+
+**Config:** none if `PUBLIC_API_URL` stays empty.
+
+### 3b. Hybrid SSR (`@astrojs/node`, `prerender = false`)
+
+Astro 5 dropped `output: 'hybrid'`. Pattern: install an adapter, keep `output: 'static'`, set `export const prerender = false` on routes that must run per request.
+
+Concrete changes:
+
+| File / unit | Change |
+|---|---|
+| `web/package.json` | Add `@astrojs/node`. `start` would become something like `node dist/server/entry.mjs` instead of `node server.mjs`. |
+| `web/astro.config.mjs` | `import node from '@astrojs/node'`; `adapter: node({ mode: 'standalone' })`. `output` can stay `'static'`. |
+| `web/src/pages/[...locale]/mints.astro` (and fedimints / lnurl-mints; maybe `index.astro`) | `export const prerender = false`. Drop `getStaticPaths` **or** keep it only if those routes stay static. SSR pages fetch `fetchMints()` **per request**. |
+| `web/server.mjs` | **Cannot stay as the only server.** Node adapter emits `dist/server/entry.mjs` plus `dist/client/` for assets. Locale 404s, ETag/cache policy, path jail, and “refuse empty root” all live in `server.mjs` today and would need reimplementation or a wrapper. |
+| `cashumints-site.service` | `ExecStart` → adapter entry; `WEB_ROOT` layout changes (`client/` vs flat dist). |
+| `cashumints-web.service` | `rsync` of `web/dist/` is wrong for `client/` + `server/`. Publish + restart model changes. A failed SSR process takes down **HTML**, not only islands. |
+| nginx | Still reverse-proxy; upstream may stay `:8789` if the adapter listens there. `proxy_read_timeout` 30s is fine for API-backed SSR. |
+
+SSR **does not** by itself refresh review **bodies** on the list (there are none). It **does** make list HTML match the API on every request, including for crawlers, and can emit real 200s for new `/mint/{host}` if that route is also `prerender = false` (today new hosts are **404** until rebuild).
+
+Per-request cost: `/mints` already does N+1 detail fetches for chips × 24 locales if you SSR all locales naively. Needs a cache or list-payload chips.
+
+### 3c. Caching layers
+
+**In-repo**
+
+| Layer | TTL | Affects |
+|---|---|---|
+| `getStats()` memo (`api/src/queries.ts`) | 60s | `/api/stats` |
+| Review / feed `SimplePool` caches in the tab | 5 min | detail reviews, `/reviews` |
+| Icons | 1 day (`max-age=86400`) | `/icons/*` |
+| `GET /api/mints`, `/api/stats`, `/api/mints/:host` | **no** `Cache-Control` from Hono (except health `no-store`) | nginx fills the gap |
+
+**nginx** (`README.md` sample; not in this repo)
+
+```nginx
+proxy_cache_path … keys_zone=cashumints:10m max_size=256m inactive=10m;
+
+location ~ ^/api/(mints|stats)(?:/|$|\?) {
+  proxy_cache cashumints;
+  proxy_cache_valid 200 30s;
+  proxy_cache_valid 404 10s;
+  proxy_cache_use_stale updating error timeout http_500 http_502 http_503;
+  proxy_cache_background_update on;
+  proxy_cache_lock on;
+}
+location /icons/ { proxy_cache …; proxy_cache_valid 200 1d; }
+location = /api/health { proxy_cache off; }
+```
+
+**HTML** from `server.mjs`: `Cache-Control: public, max-age=0, must-revalidate` + ETag. Browsers revalidate; they do not keep a 24h stale `/mints`.
+
+**Implications**
+
+- Client-side list fetch: **30s** nginx micro-cache is desirable (thundering herd). Not the 24h bug.
+- After `POST /api/index`, a new mint can be missing from `GET /api/mints` for up to 30s (404 cache 10s on unknown host). Acceptable; optional `Cache-Control: s-maxage=30` on the API to make it explicit.
+- SSR `/mints` would still sit behind that 30s API cache unless the SSR process talks to the API on loopback **bypassing nginx** (`API_URL=http://127.0.0.1:8788`), which the **build** already does. An SSR Node process on the same machine should keep using loopback, not `https://cashumints.space/api`.
+- No purge/invalidation API exists. Not needed for 30s TTL.
+
+**Rate limits:** only `POST /api/index`. `GET /api/mints` is unbounded. A list page per visitor is one small JSON payload (~tens of mints), not a risk.
+
+---
+
+## 4. Recommendation
+
+**Do 3a (client-side hydration of the list from `GET /api/mints`).** Least infrastructure change that fixes: full live mint list, live card stats, live review **counts** on the list.
+
+Do **not** move `/mints` to SSR for this goal. That replaces `server.mjs`, the publish path, and systemd, to solve a problem the API + existing island style already solve. Use SSR later only if you need **200 + HTML** for mint URLs that do not exist at build time (SEO for brand-new hosts). Those URLs already **work** for humans via the 404 resolver.
+
+### Why 3a is enough
+
+1. The API already returns the full cashu list with scores and review aggregates.
+2. Same-origin `/api` is already proxied; PulseTicker proves browser → API works in production.
+3. New mints already resolve on click (404 → `POST /api/index`). Hydrating the list is the missing half.
+4. Review **text** on `/mints` was never a list feature; card counts come from the API. Once the list refetches, counts match what MintLive/stats use (discovery-ingested). Bodies stay on the detail page via Nostr, as now.
+5. `/reviews` already has live bodies.
+
+### Effort
+
+**Small–medium, ~1–2 days** for `/mints` + fedimints + lnurl-mints + a shared card renderer. Another half day if the home top grids and “latest reviews” strip should match.
+
+Optional extra (half day): add chip fields to `GET /api/mints` so the client does not N+1.
+
+### Files that would be touched (3a)
+
+**Required for the list pages**
+
+- `web/src/pages/[...locale]/mints.astro`
+- `web/src/pages/[...locale]/fedimints.astro`
+- `web/src/pages/[...locale]/lnurl-mints.astro`
+- `web/src/lib/mint-cards.ts` (new; browser card HTML + `fetch` against `apiBase`)
+
+**Likely**
+
+- `web/src/pages/[...locale]/index.astro` (top-6 + “all N mints” count; otherwise home still lies)
+- `web/src/components/MintCard.astro` only if extracting shared helpers rather than duplicating markup
+
+**Optional API (chips)**
+
+- `shared/src/types.ts` (`MintListItem`)
+- `api/src/queries.ts` (`toListItem`)
+
+**Not required:** `astro.config.mjs`, `web/package.json` (unless you add a test), `web/server.mjs`, nginx, systemd units, `@astrojs/node`.
+
+### Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| **SEO:** crawlers see the prerendered subset | Medium if you **replace** HTML with an empty grid. Low if you **keep** build-time cards and enhance. Google does not run the same JS as users. | Keep prerendered cards. Hydration only adds/updates. `ItemList` JSON-LD stays a snapshot; acceptable. |
+| **New mint URLs return HTTP 404** until rebuild | Low for UX (resolver fills in). Medium for SEO of a mint indexed today. | Out of scope for list freshness. SSR on `mint/[host].astro` is the fix if needed. |
+| **Nostr relay latency** | Does not apply to the **list** if you use the API. Applies to `/reviews` and detail panels already (8–9s timeout, bones, retry). | Do not query relays on `/mints`. |
+| **API rate limits** | None on GET. | — |
+| **CORS** | None while `PUBLIC_API_URL` is empty (same origin). Hono already sends CORS if the API is split later. | Do not bake `API_URL` (`127.0.0.1`) into browser code. |
+| **nginx 30s cache** | Cards can lag indexing by 30s | Fine. Loopback SSR would skip it; client fetch will not. |
+| **Card markup drift** | JS builder vs `MintCard.astro` | One HTML helper, or accept a short duplication like `review-cards.ts` vs the Astro card. |
+| **View transitions** | Cards use `data-vt-*` names applied on click | Preserve those attributes in the builder. |
+| **i18n** | Card strings must use `useI18n()` in the island, not build-time `t` | Same as Reviews / PulseTicker. |
+| **Home “latest reviews”** | Still nightly unless you reuse `loadFeed` | Separate follow-up; `/reviews` is already live. |
+
+### Suggested sequence (when implementing)
+
+1. Hydrate `/mints` from `GET /api/mints?type=cashu`; keep prerendered fallback.
+2. Repeat for `/fedimints` and `/lnurl-mints`.
+3. Hydrate the home top grids (and the “all N” link) from the same endpoints.
+4. Only if chips go stale in a way users notice: extend the list payload.
+5. Revisit SSR only for mint-detail **status codes** / OG / sitemap freshness — not for the list.