Merge pull request 'Clarify the empty-WEB_ROOT hint to point at cashumints-web.' (#7) from dev into main

Reviewed-on: #7

.gitignore

@@ -2,6 +2,7 @@ node_modules/
 dist/
 .astro/
 api/data/
+deploy/
 *.log
 .DS_Store
 .env

README.md

@@ -210,7 +210,7 @@ ships at `/fonts/OFL.txt`.
 
 They sit in `src/assets/` rather than `public/` so the build fingerprints them into
 `/_astro/`, which is what makes `Cache-Control: immutable` honest and puts them under a
-cache rule the nginx config already has. Unhashed and uncached they are re-fetched on
+cache rule `web/server.mjs` already has. Unhashed and uncached they are re-fetched on
 every client-side navigation — the router re-inserts their preload links on each swap —
 and the typefaces visibly reload from page to page.
 
@@ -305,7 +305,8 @@ hashed into its filename (`mint.example.com.a1b2c3d4e5.png`) and
 is also the cache-busting story — link-preview scrapers cache an `og:image` by URL, so
 a mint whose rating moved or that went offline gets a new URL on the next scheduled
 rebuild, while the stable-named `default.png` (brand plus network stats, used by every
-non-mint page) is served with a one-hour cache instead (see the nginx block below).
+non-mint page) is served with a one-hour cache instead — the one exception in
+`cacheControl()` in `web/server.mjs`.
 
 The images are one English render shared by every locale; titles, descriptions and alt
 text translate per page. Relative times stay out of the PNGs on purpose — a "3d ago"
@@ -601,8 +602,8 @@ so a systemd `Environment=` line or a one-off `PORT=9000 pnpm dev:api` still ove
 `PUBLIC_API_URL` deliberately does **not** fall back to `API_URL`: that is a build-machine
 address, and a visitor's browser cannot reach `127.0.0.1`. Left empty, icon `src`
 attributes and island fetches are same-origin paths (`/icons/...`, `/api/...`), which the
-dev server proxies to `API_URL` and nginx forwards in production — see "Static site behind
-nginx" below.
+dev server proxies to `API_URL` and nginx forwards in production — see "nginx" under
+Deployment below.
 
 Set it only when the API answers on its own origin:
 
@@ -753,10 +754,11 @@ and capped at two with every hop re-checked, a 256KB response cap, and a per-IP
 `INDEX_RATE_LIMIT` an hour with a clean `429`. Two simultaneous submissions of one address
 share a single probe. `pnpm --filter ./api test:index` covers all of it without a network.
 
-The limit needs to be able to tell two visitors apart, so behind the nginx block below add
-`proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;` to `location /api/`. The
-last entry of that header is the one used, because it is the one the proxy vouched for;
-the header is ignored entirely when the peer is not loopback.
+The limit needs to be able to tell two visitors apart, which is what `include proxy_params;`
+on every proxied location in the nginx block below is for — it is what sets
+`X-Forwarded-For`, and without it every submission arrives from the loopback peer and
+shares one budget. The last entry of that header is the one used, because it is the one
+the proxy vouched for; the header is ignored entirely when the peer is not loopback.
 
 ### Ranking
 
@@ -780,193 +782,368 @@ literal specified behaviour. `shared/src/score.ts` carries the arithmetic, and
 
 ## Deployment
 
-### API as a systemd unit
+Three units and an nginx block. Everything this project runs listens on loopback and
+runs as the same unprivileged user; nginx terminates TLS and proxies to it, and opens no
+file belonging to the project.
+
+```
+        :443  nginx  ──►  127.0.0.1:8789   cashumints-site.service   the prerendered site
+                     └─►  127.0.0.1:8788   cashumints.service        /api/* and /icons/*
+
+                          cashumints-web.service   oneshot: build, then publish to
+                                                   /var/lib/cashumints/web
+```
+
+nginx used to point a `root` at `web/dist` instead of proxying. That is the arrangement
+to avoid, and the reason is worth stating because the failure mode is so quiet: nginx
+runs as `www-data`, everything here runs as `cashumints`, and serving files across that
+boundary requires every directory from `/` down to `dist` to be traversable by a user
+with no other business in the tree. A home directory left at its default `0700` breaks
+the whole site, and `try_files` reports a permission error as a plain miss — so the
+symptom is a blanket 404, or an internal-redirect loop that ends in a 500, with the
+cause named nowhere. Proxying removes the boundary: the process that serves the files
+is the one that built them.
+
+### Node
+
+The API and the site server both run TypeScript and ESM directly, with no build step, so
+**systemd's node must be 22.18 or newer** — that is the release where native type
+stripping stopped needing a flag. This is not the same question as `node -v` in your
+shell: a version manager puts its node on the interactive `PATH` only, while systemd
+resolves the absolute path in `ExecStart`. Check the one that matters:
+
+```bash
+/usr/bin/node --version
+```
+
+On Node 20 the API exits immediately with `ERR_UNKNOWN_FILE_EXTENSION` for `.ts` and
+restarts forever. Install Node system-wide rather than pointing `ExecStart` at a version
+manager's path, which breaks at the next upgrade and is invisible to `ProtectHome`.
+
+### The API
 
 ```ini
-# /etc/systemd/system/cashumints-api.service
+# /etc/systemd/system/cashumints.service
 [Unit]
 Description=cashumints.space indexer and API
-After=network-online.target
 Wants=network-online.target
+After=network-online.target
+# Stop after five failures in a minute rather than restarting forever: a process that
+# cannot start will not start on the 4000th attempt either, and `failed` in
+# `systemctl status` is a louder signal than a journal scrolling past. Both keys belong
+# to [Unit] — under [Service] systemd only warns and ignores them.
+StartLimitIntervalSec=60
+StartLimitBurst=5
 
 [Service]
 Type=simple
 User=cashumints
-WorkingDirectory=/srv/cashumints/api
-ExecStart=/usr/bin/node src/index.ts
+Group=cashumints
+WorkingDirectory=/home/cashumints/CashuMints.space/api
+
+# StateDirectory creates /var/lib/cashumints with the service user's ownership.
+StateDirectory=cashumints
 Environment=NODE_ENV=production
-Environment=PORT=8787
+Environment=PORT=8788
 Environment=DB_PATH=/var/lib/cashumints/cashumints.db
 Environment=ICON_DIR=/var/lib/cashumints/icons
-# For Postgres, replace DB_PATH with DATABASE_URL and add After=postgresql.service above.
+# For Postgres, replace DB_PATH with DATABASE_URL and add After=postgresql.service.
 # Keep ICON_DIR either way: cached icons are files, not rows.
-Restart=always
-RestartSec=5
+ExecStart=/usr/bin/node --env-file-if-exists=../.env src/index.ts
+
+Restart=on-failure
+RestartSec=5s
 # The process finishes its in-flight probe batch and closes the database on SIGTERM.
 KillSignal=SIGTERM
-TimeoutStopSec=30
+TimeoutStopSec=30s
+UMask=0027
 
 NoNewPrivileges=true
 PrivateTmp=true
+PrivateDevices=true
 ProtectSystem=strict
-ProtectHome=true
+ProtectHome=read-only
 ReadWritePaths=/var/lib/cashumints
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
 
 [Install]
 WantedBy=multi-user.target
 ```
 
-```bash
-sudo systemctl enable --now cashumints-api
+`Environment=` in the unit and `--env-file-if-exists=../.env` are not interchangeable.
+Anything set in the unit wins, so a value that must not drift with an edit to `.env`
+belongs in the unit; a secret belongs in `.env`, which is not world-readable.
+
+### The site server
+
+`web/server.mjs` serves the prerendered tree and nothing else — no template, no database
+handle, no route table. It is a few hundred lines of `node:http` with no dependencies,
+and `pnpm --filter ./web test` covers what it has to get right: a miss answers 404
+rather than 200 with a 404-shaped body, a miss under a locale stays in that locale,
+hashed assets are immutable and markup is not, and nothing outside the root is readable
+however the path is spelled.
+
+Two behaviours are worth knowing about because they are load-bearing:
+
+- **It refuses to start on a root with no `index.html`.** The failure that prevents is a
+  process that starts cleanly, answers every request with 404 and looks healthy to
+  anything watching the port.
+- **It does not serve `web/dist`.** `astro build` empties `dist` before it writes, so
+  serving it directly means a rebuild takes the site down for the length of the build.
+  It serves the published copy at `WEB_ROOT` instead.
+
+```ini
+# /etc/systemd/system/cashumints-site.service
+[Unit]
+Description=cashumints.space static site server
+Wants=network-online.target
+After=network-online.target
+StartLimitIntervalSec=60
+StartLimitBurst=5
+# Not Requires=cashumints.service: the pages are prerendered, so the site keeps serving
+# a correct-as-of-last-build copy while the API is down. Only the islands go quiet.
+
+[Service]
+Type=simple
+User=cashumints
+Group=cashumints
+WorkingDirectory=/home/cashumints/CashuMints.space/web
+
+StateDirectory=cashumints
+Environment=NODE_ENV=production
+Environment=SITE_PORT=8789
+Environment=SITE_HOST=127.0.0.1
+Environment=WEB_ROOT=/var/lib/cashumints/web
+ExecStart=/usr/bin/node server.mjs
+
+Restart=on-failure
+RestartSec=5s
+KillSignal=SIGTERM
+TimeoutStopSec=15s
+
+NoNewPrivileges=true
+PrivateTmp=true
+PrivateDevices=true
+ProtectSystem=strict
+# Read-only rather than absent: server.mjs itself lives under /home/cashumints.
+ProtectHome=read-only
+ReadWritePaths=/var/lib/cashumints
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+RestrictSUIDSGID=true
+LockPersonality=true
+
+[Install]
+WantedBy=multi-user.target
 ```
 
-### Static site behind nginx
+### nginx
 
-```nginx
-server {
-  listen 443 ssl http2;
-  server_name cashumints.space;
-
-  root /srv/cashumints/web;
-  index index.html;
-
-  # Astro emits directory-style routes, so try the directory index before 404.
-  location / {
-    try_files $uri $uri/ $uri.html /404.html;
-  }
-
-  # A miss under a language prefix answers in that language. Without these two blocks
-  # every 404 is the English one, including the ones a Spanish reader reaches by
-  # following a Spanish link, and the language control on it would take them to a page
-  # they were not on.
-  location /es/ {
-    try_files $uri $uri/ $uri.html /es/404/index.html;
-  }
-
-  location /nl/ {
-    try_files $uri $uri/ $uri.html /nl/404/index.html;
-  }
-
-  # Fingerprinted assets are immutable. This covers the CSS, the island bundles and the
-  # webfonts, which are all emitted here with a content hash in the name. Serving the
-  # fonts without it is visible, not theoretical: the view transition router re-inserts
-  # the font preload links on every navigation, so an uncached font is re-fetched on
-  # each page change and the typefaces flicker as they reload.
-  location /_astro/ {
-    expires 1y;
-    add_header Cache-Control "public, immutable";
-  }
-
-  # The favicons and the manifest. Not fingerprinted, since their names are referenced
-  # from outside the site, so a week with revalidation rather than a year of immutability.
-  location ~* ^/(og\.png|favicon\.(ico|svg)|apple-touch-icon\.png|icon-\d+\.png|site\.webmanifest)$ {
-    expires 7d;
-    add_header Cache-Control "public";
-  }
-
-  # The generated social cards (`pnpm og`). The default card keeps a stable name and
-  # changes with the site's stats, so it gets an hour; every per-mint card carries a
-  # content hash in its filename and a stale hash is never referenced again, so those
-  # are immutable. (The hash is also what actually refreshes link previews: the big
-  # scrapers cache an og:image by URL and ignore these headers.)
-  location = /og/default.png {
-    add_header Cache-Control "public, max-age=3600";
-  }
-
-  location /og/ {
-    expires 1y;
-    add_header Cache-Control "public, immutable";
-  }
-
-  # Same-origin API and icons, matching the default empty PUBLIC_API_URL. Drop these two
-  # blocks only if you build with PUBLIC_API_URL pointing at a separate API host.
-  location /api/ {
-    proxy_pass http://127.0.0.1:8787;
-    # POST /api/index is rate limited per address, and without this every visitor
-    # arrives as 127.0.0.1 and shares one budget.
-    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-  }
-
-  location /icons/ {
-    proxy_pass http://127.0.0.1:8787;
-    expires 1d;
-  }
-
-  error_page 404 /404.html;
-}
-```
-
-Add a `location` block per language when you add one. The 404 is also the client-side
-fallback for a mint discovered since the last build, and it reads the locale off its own
-URL, so `/es/mint/some-new-mint` resolves that mint and renders its summary in Spanish.
-
-### API behind nginx, with a micro-cache
-
-`/api/mints` and `/api/stats` change at most every few minutes but can be requested by
-every visitor at once. A short micro-cache absorbs that without making the data stale.
-`/api/health` is deliberately excluded: it is the endpoint you page on.
+The whole public surface. Two upstreams, no `root`, no `try_files`, no `error_page`: a
+miss is the site server's 404 page, in the right language and with a 404 status, and an
+nginx error page here would replace it with a blank one and hide which upstream failed.
 
 ```nginx
 proxy_cache_path /var/cache/nginx/cashumints levels=1:2 keys_zone=cashumints:10m
                  max_size=256m inactive=10m use_temp_path=off;
 
+# Keep a few connections open to each upstream rather than reconnecting per request.
+# Both processes hold idle sockets longer than nginx does, so nginx is always the side
+# that closes — the reverse of that race is what produces sporadic 502s under load.
+upstream cashumints_site { server 127.0.0.1:8789; keepalive 16; }
+upstream cashumints_api  { server 127.0.0.1:8788; keepalive 8; }
+
 server {
   listen 443 ssl http2;
-  server_name api.cashumints.space;
+  listen [::]:443 ssl http2;
+  server_name cashumints.space;
 
-  location /api/health {
-    proxy_pass http://127.0.0.1:8787;
-    proxy_cache off;
-    add_header Cache-Control "no-store" always;
+  ssl_certificate /etc/letsencrypt/live/cashumints.space/fullchain.pem;
+  ssl_certificate_key /etc/letsencrypt/live/cashumints.space/privkey.pem;
+  include /etc/letsencrypt/options-ssl-nginx.conf;
+  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
+
+  # Set here because this is the only part of the stack that knows a request arrived
+  # over TLS. Note that add_header replaces rather than merges: any location declaring
+  # its own add_header drops this one and has to restate it.
+  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+
+  # The upstreams send no Content-Encoding, so compression is nginx's to do.
+  # gzip_proxied any is required — without it nginx will not compress a proxied response
+  # at all. The home page goes out at a fifth of its size.
+  gzip on;
+  gzip_proxied any;
+  gzip_vary on;
+  gzip_comp_level 5;
+  gzip_min_length 1024;
+  gzip_types text/plain text/css text/javascript application/javascript application/json
+             application/manifest+json application/xml image/svg+xml;
+
+  # Nothing here accepts an upload; the API's largest body is an 8 KB JSON submission.
+  client_max_body_size 16k;
+
+  # Both upstreams are a process on this machine. A slow response is a bug, not a
+  # network condition, and failing fast beats holding a worker for a minute.
+  proxy_connect_timeout 2s;
+  proxy_read_timeout 30s;
+  proxy_send_timeout 30s;
+
+  # HTTP/1.1 with an empty Connection header is what makes the keepalive pools work;
+  # the default 1.0 opens a new socket per request.
+  proxy_http_version 1.1;
+  proxy_set_header Connection "";
+
+  # proxy_params (Debian) sets Host, X-Real-IP, X-Forwarded-For and X-Forwarded-Proto.
+  # That include is load-bearing: the API's rate limiter reads the last hop of
+  # X-Forwarded-For to tell two visitors apart, and without it every request arrives from
+  # the loopback peer and shares one bucket. Do not also set X-Forwarded-For beside the
+  # include — declaring it twice is what produces nginx's proxy_headers_hash warning.
+
+  # Cache-Control comes from server.mjs: a year and immutable for anything with a
+  # content hash in its name, revalidate-every-time for markup. Nothing to restate here.
+  location / {
+    proxy_pass http://cashumints_site;
+    include proxy_params;
   }
 
-  location ~ ^/api/(mints|stats) {
-    proxy_pass http://127.0.0.1:8787;
+  # Health must always reflect the live process. It is the endpoint you page on.
+  location = /api/health {
+    proxy_pass http://cashumints_api;
+    proxy_cache off;
+    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+    add_header Cache-Control "no-store" always;
+    include proxy_params;
+  }
 
+  # /api/mints and /api/stats change every few minutes but can be asked for by every
+  # visitor at once. A micro-cache absorbs that without making the data stale.
+  location ~ ^/api/(mints|stats)(?:/|$|\?) {
+    proxy_pass http://cashumints_api;
     proxy_cache cashumints;
     proxy_cache_valid 200 30s;
     proxy_cache_valid 404 10s;
-    # Serve the previous response while one request refreshes it, so a slow
-    # backend never becomes a slow page.
+    # Serve the previous response while one request refreshes it, so a slow backend
+    # never becomes a slow page.
     proxy_cache_use_stale updating error timeout http_500 http_502 http_503;
     proxy_cache_background_update on;
     proxy_cache_lock on;
+    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
     add_header X-Cache-Status $upstream_cache_status always;
+    include proxy_params;
+  }
+
+  location /api/ {
+    proxy_pass http://cashumints_api;
+    include proxy_params;
   }
 
   location /icons/ {
-    proxy_pass http://127.0.0.1:8787;
+    proxy_pass http://cashumints_api;
     proxy_cache cashumints;
     proxy_cache_valid 200 1d;
+    include proxy_params;
   }
 }
+
+server {
+  listen 80;
+  listen [::]:80;
+  server_name cashumints.space;
+
+  location /.well-known/acme-challenge/ { root /var/www/html; }
+  location / { return 301 https://cashumints.space$request_uri; }
+}
 ```
 
+`/api/` and `/icons/` are proxied same-origin because `PUBLIC_API_URL` is empty, which is
+the default. Drop those blocks only if you build with it pointing at a separate API host.
+
+nginx 1.25 and later want `http2 on;` on its own line and warn about the `listen … http2`
+form above; Debian 12 ships 1.22, where the newer form is an unknown directive. The form
+above is the one that works on both.
+
 ### Rebuilds
 
 Mint pages are prerendered, so new mints and new review counts appear at the next build.
-A nightly rebuild is enough; the site stays correct in between because the islands refresh
-status and reviews at runtime, and an unbuilt mint still resolves through the client-side
-fallback on the 404 page.
+A nightly rebuild is enough; the site stays correct in between because the islands
+refresh status and reviews at runtime, and an unbuilt mint still resolves through the
+client-side fallback on the 404 page.
+
+Publishing is a separate step from building, and the separation is the point: the copy
+the site server reads is only touched once a build has succeeded, so a failed build
+leaves the previous site up rather than replacing it with a half-written one.
 
 ```ini
-# /etc/systemd/system/cashumints-build.service
+# /etc/systemd/system/cashumints-web.service
 [Unit]
 Description=Rebuild the cashumints.space static site
+# Every page's data comes from the API over loopback, so the API has to be up.
+# Requires= rather than Wants=: a dead API should abort the build, not replace a good
+# site with an empty one.
+Requires=cashumints.service
+After=cashumints.service network-online.target
+Wants=network-online.target
 
 [Service]
 Type=oneshot
 User=cashumints
-WorkingDirectory=/srv/cashumints
-Environment=API_URL=http://127.0.0.1:8787
-Environment=PUBLIC_API_URL=https://api.cashumints.space
+Group=cashumints
+WorkingDirectory=/home/cashumints/CashuMints.space
+StateDirectory=cashumints
+
+Environment=NODE_ENV=production
+# Where the build reaches the API. Must match PORT= in cashumints.service.
+Environment=API_URL=http://127.0.0.1:8788
 Environment=SITE_URL=https://cashumints.space
+# Browser-facing origin. Empty means same origin: islands fetch /api/... and nginx
+# forwards it. Declared even though it is empty, because systemd's environment wins over
+# .env — so what a production build emits cannot drift with an edit to that file.
+Environment=PUBLIC_API_URL=
+
+# After= orders the start; it does not wait for the port to accept connections. At boot
+# the API is still opening its database and probing, so block until it reports healthy
+# rather than letting the first fetch die on ECONNREFUSED. /api/health answers 503 until
+# it is genuinely ready, and curl -f treats that as a failure, so the loop keeps waiting.
+ExecStartPre=/usr/bin/timeout 90 /bin/sh -c 'until curl -sf -o /dev/null http://127.0.0.1:8788/api/health; do sleep 1; done'
+# Check `which pnpm` on the host: a corepack or pnpm-home install sits outside /usr/bin,
+# and systemd's PATH does not include it.
 ExecStart=/usr/bin/pnpm build
-ExecStartPost=/usr/bin/rsync -a --delete web/dist/ /srv/cashumints/web/
+# --delay-updates stages the changed files and renames them in at the end, so the window
+# where the tree is a mix of two builds is a rename rather than a whole transfer, and
+# --delete-after keeps removals from landing before their replacements. Unchanged files
+# — every hashed asset and card, which is nearly all of it — are not touched at all.
+ExecStartPost=/usr/bin/rsync -a --delete-after --delay-updates web/dist/ /var/lib/cashumints/web/
+
+# ~500 prerendered pages plus a card per mint. Minutes, not seconds, on a small VPS, and
+# TimeoutStartSec is what bounds a Type=oneshot.
+TimeoutStartSec=1800
+# A nightly rebuild should not starve the API it is reading from.
+Nice=10
+UMask=0022
+
+NoNewPrivileges=true
+PrivateTmp=true
+PrivateDevices=true
+# ProtectHome is deliberately absent, unlike in the other two units: this one writes
+# inside /home/cashumints — web/dist, web/public/og, web/src/generated and the pnpm
+# store are all under it.
+ProtectSystem=full
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
 ```
 
+There is deliberately no `[Install]` section: a rebuild should be scheduled, not fired on
+every boot.
+
 ```ini
-# /etc/systemd/system/cashumints-build.timer
+# /etc/systemd/system/cashumints-web.timer
 [Unit]
 Description=Nightly cashumints.space rebuild
 
@@ -978,10 +1155,31 @@ Persistent=true
 WantedBy=timers.target
 ```
 
+### First deploy
+
+Order matters once: the site server refuses to start against a root that has no
+`index.html`, so the build has to publish before it comes up.
+
 ```bash
-sudo systemctl enable --now cashumints-build.timer
+/usr/bin/node --version                        # 22.18 or newer, or the API will not run
+sudo systemctl enable --now cashumints         # API first: the build reads from it
+sudo systemctl start cashumints-web            # build, then publish to /var/lib/cashumints/web
+sudo systemctl enable --now cashumints-site    # now it has something to serve
+sudo systemctl enable --now cashumints-web.timer
+sudo nginx -t && sudo systemctl reload nginx
 ```
 
+```bash
+curl -sI https://cashumints.space | head -1              # 200
+curl -sI https://cashumints.space/nope | head -1         # 404, not 200
+curl -s  https://cashumints.space/api/health             # status ok
+```
+
+A 502 on `/` means the site server is down or was never started; a 502 on `/api/` means
+the API is. They fail independently, which is the other thing proxying buys: the
+prerendered site keeps serving while the API is restarting.
+
+
 ## Licence
 
 See `LICENSE`.

package.json

@@ -13,6 +13,7 @@
     "seed": "pnpm --filter ./api seed",
     "bones": "pnpm --filter ./web bones",
     "build": "pnpm --filter ./shared build && pnpm --filter ./web build",
+    "start": "pnpm --filter ./web start",
     "check:links": "pnpm --filter ./web check:links",
     "typecheck": "pnpm -r typecheck",
     "check:i18n": "pnpm --filter ./web check:i18n",

web/package.json

@@ -8,6 +8,7 @@
     "og": "node scripts/og/build-og.mjs",
     "og:fixtures": "node scripts/og/build-og.mjs --fixtures",
     "build": "node scripts/og/build-og.mjs && astro build",
+    "start": "node server.mjs",
     "preview": "astro preview",
     "typecheck": "astro check",
     "bones": "node --env-file-if-exists=../.env scripts/bones.mjs",

web/server.mjs

@@ -0,0 +1,386 @@
+/**
+ * The production web server.
+ *
+ * The site is `output: 'static'`: `pnpm build` prerenders every page and this process
+ * only hands the result out over HTTP. nginx sits in front of it and proxies, rather
+ * than pointing a `root` at the built tree, so that nothing outside this file decides
+ * what is readable. That is not a stylistic preference — it removes two whole classes
+ * of failure the file-serving arrangement kept producing:
+ *
+ *   traversal permissions   nginx runs as www-data and the build runs as cashumints,
+ *                           so every directory from / down to dist had to be traversable
+ *                           by a user with no other business in it. One 0700 home
+ *                           directory anywhere in the chain took the site down, and
+ *                           `try_files` reports a permission error as a plain miss, so
+ *                           the symptom was a blanket 404 or an internal-redirect loop
+ *                           ending in 500 — never the actual cause.
+ *
+ *   duplicated routing      the locale 404 rule lived in nginx as a hand-maintained
+ *                           alternation of 23 codes. Adding a language meant editing a
+ *                           file that is not in this repository, and forgetting to was
+ *                           silent. Locale 404s are resolved by looking in the built
+ *                           tree now, so the list cannot drift.
+ *
+ * Reading files is all it does. There is no template, no database handle and no route
+ * table: `/api/*` and `/icons/*` belong to the API on its own port and nginx forwards
+ * them there directly.
+ *
+ * Env:
+ *   SITE_PORT   8789         port to listen on
+ *   SITE_HOST   127.0.0.1    interface to bind; loopback because nginx terminates TLS
+ *   WEB_ROOT    ./dist       the tree to serve
+ */
+import fs from 'node:fs';
+import fsp from 'node:fs/promises';
+import http from 'node:http';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+
+function int(raw, fallback) {
+  const n = Number.parseInt(raw ?? '', 10);
+  return Number.isFinite(n) && n > 0 ? n : fallback;
+}
+
+/**
+ * The tree to serve, absolute.
+ *
+ * Defaults to the build output next to this file, which is what `pnpm start` and the
+ * tests use. In production it points at a copy outside the checkout: `pnpm build`
+ * empties dist before it writes, so serving dist directly means a rebuild takes the
+ * whole site down for the length of the build. See cashumints-web.service.
+ */
+export const ROOT = path.resolve(process.env.WEB_ROOT ?? path.join(here, 'dist'));
+
+const PORT = int(process.env.SITE_PORT, 8789);
+const HOST = process.env.SITE_HOST ?? '127.0.0.1';
+
+/** One line per event, key=value after the message. Matches the API's log format. */
+function log(level, msg, fields = {}) {
+  const parts = [new Date().toISOString(), level.toUpperCase(), msg];
+  for (const [k, v] of Object.entries(fields)) {
+    if (v !== undefined && v !== null) parts.push(`${k}=${v}`);
+  }
+  const line = parts.join(' ');
+  if (level === 'error') console.error(line);
+  else console.log(line);
+}
+
+const TYPES = new Map(Object.entries({
+  '.html': 'text/html; charset=utf-8',
+  '.js': 'text/javascript; charset=utf-8',
+  '.mjs': 'text/javascript; charset=utf-8',
+  '.css': 'text/css; charset=utf-8',
+  '.json': 'application/json; charset=utf-8',
+  '.map': 'application/json; charset=utf-8',
+  '.webmanifest': 'application/manifest+json; charset=utf-8',
+  '.xml': 'application/xml; charset=utf-8',
+  '.txt': 'text/plain; charset=utf-8',
+  '.svg': 'image/svg+xml',
+  '.png': 'image/png',
+  '.jpg': 'image/jpeg',
+  '.jpeg': 'image/jpeg',
+  '.webp': 'image/webp',
+  '.avif': 'image/avif',
+  '.gif': 'image/gif',
+  '.ico': 'image/x-icon',
+  '.woff2': 'font/woff2',
+  '.woff': 'font/woff',
+  '.ttf': 'font/ttf',
+  '.wasm': 'application/wasm',
+}));
+
+const IMMUTABLE = 'public, max-age=31536000, immutable';
+const WEEK = 'public, max-age=604800';
+const HOUR = 'public, max-age=3600';
+/** Zero lifetime but cacheable: the client keeps the body and revalidates into a 304. */
+const REVALIDATE = 'public, max-age=0, must-revalidate';
+
+/**
+ * How long a response may be reused.
+ *
+ * Everything Vite and the card builder emit carries a content hash in its filename, so
+ * those are immutable for a year — a changed file is a changed URL. Markup is the
+ * opposite: the URLs are permanent and the bytes change on every rebuild, so it
+ * revalidates every time and the ETag below turns that into a 304 in the usual case.
+ *
+ * `/og/default.png` is the one unhashed card, served for pages that have no mint of
+ * their own, so it gets an hour rather than a year.
+ */
+export function cacheControl(urlPath, ext) {
+  if (ext === '.html') return REVALIDATE;
+  if (urlPath === '/og/default.png') return HOUR;
+  if (urlPath.startsWith('/_astro/') || urlPath.startsWith('/og/')) return IMMUTABLE;
+  if (urlPath === '/robots.txt' || urlPath === '/sitemap.xml') return HOUR;
+  return WEEK;
+}
+
+/**
+ * Turn a request path into a path inside ROOT, or null if it escapes or is malformed.
+ *
+ * Returns the *relative* path so the caller can join it against ROOT; every segment is
+ * checked rather than trusting `path.join` to have swallowed the `..`. Dotfiles are
+ * refused outright: a static build emits none, so a request for one is either a probe
+ * or a mistake, and neither should be answered with bytes.
+ */
+export function safePath(urlPath) {
+  if (urlPath.includes('\0')) return null;
+
+  const normalized = path.posix.normalize(urlPath);
+  if (!normalized.startsWith('/')) return null;
+
+  const segments = normalized.split('/').filter(Boolean);
+  for (const segment of segments) {
+    if (segment === '..' || segment.startsWith('.')) return null;
+  }
+  return segments;
+}
+
+async function statFile(file) {
+  try {
+    const stats = await fsp.stat(file);
+    return stats.isFile() ? stats : null;
+  } catch {
+    return null;
+  }
+}
+
+/**
+ * The candidates for one request path, in order.
+ *
+ * Astro emits directory-style routes — /mints is dist/mints/index.html — and
+ * `trailingSlash: 'ignore'` means /mints and /mints/ are both the page. Trying the
+ * literal path first keeps assets a single stat; the `.html` candidate covers the flat
+ * files at the root, /404.html among them.
+ */
+export function candidates(segments) {
+  const rel = segments.join('/');
+  if (rel === '') return ['index.html'];
+  return [rel, `${rel}/index.html`, `${rel}.html`];
+}
+
+/**
+ * The 404 body for a path, and it is not always the English one.
+ *
+ * A miss under /es keeps the visitor on the Spanish 404 rather than bouncing them into
+ * English, which is the same reason `redirectToDefaultLocale` is false in the Astro
+ * config. Which prefixes count is decided by what the build actually emitted: if
+ * <prefix>/404/index.html exists, the prefix is a locale. Nothing to keep in sync.
+ */
+async function notFoundBody(segments) {
+  const first = segments[0];
+  if (first) {
+    const localized = path.join(ROOT, first, '404', 'index.html');
+    const stats = await statFile(localized);
+    if (stats) return { file: localized, stats };
+  }
+  const fallback = path.join(ROOT, '404.html');
+  const stats = await statFile(fallback);
+  return stats ? { file: fallback, stats } : null;
+}
+
+/** nginx's ETag format: hex mtime and hex size, strong. Cheap, and changes on rebuild. */
+function etagFor(stats) {
+  return `"${Math.floor(stats.mtimeMs / 1000).toString(16)}-${stats.size.toString(16)}"`;
+}
+
+/** RFC 9110: a list of entity tags, or `*`. Weak comparison is right for GET. */
+function etagMatches(header, etag) {
+  if (!header) return false;
+  if (header.trim() === '*') return true;
+  const bare = etag.replace(/^W\//, '');
+  return header
+    .split(',')
+    .map((candidate) => candidate.trim().replace(/^W\//, ''))
+    .includes(bare);
+}
+
+function notModified(req, etag, lastModified) {
+  if (etagMatches(req.headers['if-none-match'], etag)) return true;
+
+  // Only consulted when the client sent no ETag, per RFC 9110 §13.1.3.
+  if (req.headers['if-none-match']) return false;
+  const since = Date.parse(req.headers['if-modified-since'] ?? '');
+  return Number.isFinite(since) && Math.floor(lastModified / 1000) * 1000 <= since;
+}
+
+/**
+ * Headers every response carries.
+ *
+ * No CSP here on purpose. The site loads an analytics script from another origin, the
+ * review islands open websockets to whatever relays are configured and the status
+ * islands fetch whatever mint URLs the index holds, so a policy tight enough to be
+ * worth having has to be derived from those lists rather than guessed at — and a wrong
+ * one fails as a silently broken island. HSTS belongs to nginx, which is what actually
+ * terminates TLS.
+ */
+function baseHeaders() {
+  return {
+    'X-Content-Type-Options': 'nosniff',
+    'Referrer-Policy': 'strict-origin-when-cross-origin',
+    'X-Frame-Options': 'DENY',
+  };
+}
+
+function send(res, status, headers, body) {
+  res.writeHead(status, { ...baseHeaders(), ...headers });
+  res.end(body);
+}
+
+/** Stream a file, or just its headers for HEAD. */
+function sendFile(req, res, status, file, stats, urlPath) {
+  const ext = path.extname(file).toLowerCase();
+  const etag = etagFor(stats);
+
+  const headers = {
+    'Content-Type': TYPES.get(ext) ?? 'application/octet-stream',
+    'Content-Length': stats.size,
+    'Last-Modified': new Date(stats.mtimeMs).toUTCString(),
+    ETag: etag,
+    'Cache-Control': cacheControl(urlPath, ext),
+    ...baseHeaders(),
+  };
+
+  // A 304 must not carry a body or a Content-Length describing one.
+  if (status === 200 && notModified(req, etag, stats.mtimeMs)) {
+    delete headers['Content-Length'];
+    delete headers['Content-Type'];
+    res.writeHead(304, headers);
+    res.end();
+    return;
+  }
+
+  res.writeHead(status, headers);
+  if (req.method === 'HEAD') {
+    res.end();
+    return;
+  }
+
+  const stream = fs.createReadStream(file);
+  stream.on('error', (err) => {
+    log('error', 'read failed', { file, err: err.message });
+    res.destroy();
+  });
+  // Kill the read when the client hangs up mid-transfer rather than draining the file.
+  res.on('close', () => stream.destroy());
+  stream.pipe(res);
+}
+
+async function handle(req, res) {
+  if (req.method !== 'GET' && req.method !== 'HEAD') {
+    send(res, 405, { Allow: 'GET, HEAD', 'Content-Type': 'text/plain; charset=utf-8' }, 'Method Not Allowed\n');
+    return;
+  }
+
+  let urlPath;
+  try {
+    urlPath = decodeURIComponent(new URL(req.url, 'http://localhost').pathname);
+  } catch {
+    send(res, 400, { 'Content-Type': 'text/plain; charset=utf-8' }, 'Bad Request\n');
+    return;
+  }
+
+  const segments = safePath(urlPath);
+  if (!segments) {
+    send(res, 400, { 'Content-Type': 'text/plain; charset=utf-8' }, 'Bad Request\n');
+    return;
+  }
+
+  for (const candidate of candidates(segments)) {
+    const file = path.join(ROOT, candidate);
+    // Belt and braces: safePath already refused `..`, this refuses anything that still
+    // resolved outside the tree, a symlink in the build output included.
+    if (file !== ROOT && !file.startsWith(ROOT + path.sep)) break;
+
+    const stats = await statFile(file);
+    if (stats) {
+      sendFile(req, res, 200, file, stats, urlPath);
+      return;
+    }
+  }
+
+  const miss = await notFoundBody(segments);
+  if (!miss) {
+    send(res, 404, { 'Content-Type': 'text/plain; charset=utf-8', 'Cache-Control': REVALIDATE }, 'Not Found\n');
+    return;
+  }
+  // Served as a 404, not a 200 with a 404-shaped body: a soft 404 gets every typo'd
+  // URL indexed as a real page.
+  sendFile(req, res, 404, miss.file, miss.stats, urlPath);
+}
+
+export function createServer() {
+  const server = http.createServer((req, res) => {
+    handle(req, res).catch((err) => {
+      log('error', 'request failed', { path: req.url, err: err.message });
+      if (!res.headersSent) {
+        send(res, 500, { 'Content-Type': 'text/plain; charset=utf-8', 'Cache-Control': 'no-store' }, 'Internal Server Error\n');
+      } else {
+        res.destroy();
+      }
+    });
+  });
+
+  /*
+   * Longer than nginx's upstream keepalive, and headersTimeout longer still.
+   *
+   * If this end closes an idle connection at the same moment nginx reuses it, nginx has
+   * nothing to retry and reports 502. Outlasting the proxy makes the proxy always the
+   * one to close, which is the race-free direction.
+   */
+  server.keepAliveTimeout = 65_000;
+  server.headersTimeout = 66_000;
+
+  // A malformed request line should not take the process with it.
+  server.on('clientError', (err, socket) => {
+    if (err.code === 'ECONNRESET' || !socket.writable) return;
+    socket.end('HTTP/1.1 400 Bad Request\r\nConnection: close\r\n\r\n');
+  });
+
+  return server;
+}
+
+/**
+ * Refuse to start on an empty or unreadable root.
+ *
+ * The failure this prevents is the one that is hardest to see: a process that starts
+ * cleanly, answers every request with a 404 and looks healthy to anything watching the
+ * port. Exiting non-zero puts the reason in `systemctl status` instead.
+ */
+async function checkRoot() {
+  const index = path.join(ROOT, 'index.html');
+  if (await statFile(index)) return;
+  log('error', 'web root has no index.html', {
+    root: ROOT,
+    hint: 'a manual `pnpm build` only writes web/dist; `systemctl start cashumints-web` builds and publishes to WEB_ROOT',
+  });
+  process.exit(1);
+}
+
+/** Only when run directly, so the tests can import the pieces above. */
+if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  await checkRoot();
+
+  const server = createServer();
+  server.listen(PORT, HOST, () => {
+    log('info', 'site listening', { host: HOST, port: PORT, root: ROOT });
+  });
+
+  let shuttingDown = false;
+  for (const signal of ['SIGTERM', 'SIGINT']) {
+    process.on(signal, () => {
+      if (shuttingDown) return;
+      shuttingDown = true;
+      log('info', 'shutting down', { signal });
+
+      server.close(() => process.exit(0));
+      // Idle keep-alive connections would otherwise hold the close open for a minute.
+      server.closeIdleConnections();
+      setTimeout(() => {
+        server.closeAllConnections();
+        process.exit(0);
+      }, 10_000).unref();
+    });
+  }
+}

web/test/server.test.mjs

@@ -0,0 +1,166 @@
+/**
+ * What the production server has to get right that a static file server does not.
+ *
+ * The site was served by nginx pointing a `root` at dist until this process took over,
+ * and the rules that lived in that config are the ones worth pinning down here: a miss
+ * has to answer 404 rather than 200 with a 404-shaped page, a miss under a locale has
+ * to stay in that locale, hashed assets have to be immutable and markup must not be,
+ * and nothing outside the root may be readable however the path is spelled.
+ *
+ * A fixture tree rather than dist/: these are assertions about the server, and running
+ * them should not require a build.
+ */
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs/promises';
+import os from 'node:os';
+import path from 'node:path';
+
+const root = await fs.mkdtemp(path.join(os.tmpdir(), 'cashumints-web-'));
+
+const write = async (rel, body) => {
+  await fs.mkdir(path.join(root, path.dirname(rel)), { recursive: true });
+  await fs.writeFile(path.join(root, rel), body);
+};
+
+await write('index.html', '<!doctype html><html lang="en">home</html>');
+await write('404.html', '<!doctype html><html lang="en">missing</html>');
+await write('es/404/index.html', '<!doctype html><html lang="es">no encontrado</html>');
+await write('es/mints/index.html', '<!doctype html><html lang="es">casas</html>');
+await write('mints/index.html', '<!doctype html><html lang="en">mints</html>');
+await write('robots.txt', 'User-agent: *\n');
+await write('_astro/app.abc123.js', 'export default 1;\n');
+await write('og/default.png', 'not really a png');
+await write('og/mint.abc123.png', 'not really a png either');
+
+// ROOT is read from the environment once, at import.
+process.env.WEB_ROOT = root;
+const { createServer, safePath, candidates, cacheControl } = await import('../server.mjs');
+
+const server = createServer();
+await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
+const base = `http://127.0.0.1:${server.address().port}`;
+
+test.after(async () => {
+  server.closeAllConnections();
+  await new Promise((resolve) => server.close(resolve));
+  await fs.rm(root, { recursive: true, force: true });
+});
+
+const get = (p, init) => fetch(`${base}${p}`, init);
+
+test('serves the index at the root', async () => {
+  const res = await get('/');
+  assert.equal(res.status, 200);
+  assert.equal(res.headers.get('content-type'), 'text/html; charset=utf-8');
+  assert.match(await res.text(), /home/);
+});
+
+test('resolves directory routes with and without a trailing slash', async () => {
+  for (const p of ['/mints', '/mints/']) {
+    const res = await get(p);
+    assert.equal(res.status, 200, p);
+    assert.match(await res.text(), /mints/);
+  }
+});
+
+test('a miss is a 404, not a 200 carrying the 404 page', async () => {
+  const res = await get('/nope');
+  assert.equal(res.status, 404);
+  assert.match(await res.text(), /missing/);
+});
+
+test('a miss under a locale stays in that locale', async () => {
+  const res = await get('/es/nope');
+  assert.equal(res.status, 404);
+  assert.match(await res.text(), /no encontrado/);
+});
+
+test('a miss under a prefix that is not a locale falls back to English', async () => {
+  const res = await get('/mint/does-not-exist');
+  assert.equal(res.status, 404);
+  assert.match(await res.text(), /missing/);
+});
+
+test('hashed assets are immutable and markup is not', async () => {
+  const asset = await get('/_astro/app.abc123.js');
+  assert.equal(asset.headers.get('cache-control'), 'public, max-age=31536000, immutable');
+
+  const page = await get('/');
+  assert.equal(page.headers.get('cache-control'), 'public, max-age=0, must-revalidate');
+});
+
+test('the one unhashed card is not cached for a year', async () => {
+  const shared = await get('/og/default.png');
+  assert.equal(shared.headers.get('cache-control'), 'public, max-age=3600');
+
+  const hashed = await get('/og/mint.abc123.png');
+  assert.equal(hashed.headers.get('cache-control'), 'public, max-age=31536000, immutable');
+});
+
+test('a matching ETag revalidates into a bodiless 304', async () => {
+  const first = await get('/');
+  const etag = first.headers.get('etag');
+  assert.ok(etag);
+
+  const second = await get('/', { headers: { 'If-None-Match': etag } });
+  assert.equal(second.status, 304);
+  assert.equal(await second.text(), '');
+});
+
+test('HEAD answers with the headers and no body', async () => {
+  const res = await get('/', { method: 'HEAD' });
+  assert.equal(res.status, 200);
+  assert.equal(res.headers.get('content-length'), String((await fs.stat(path.join(root, 'index.html'))).size));
+  assert.equal(await res.text(), '');
+});
+
+test('anything but GET and HEAD is refused', async () => {
+  const res = await get('/', { method: 'POST' });
+  assert.equal(res.status, 405);
+  assert.equal(res.headers.get('allow'), 'GET, HEAD');
+});
+
+test('nothing outside the root is readable', async () => {
+  await fs.writeFile(path.join(root, '..', 'cashumints-secret.txt'), 'secret');
+
+  for (const p of ['/../cashumints-secret.txt', '/%2e%2e/cashumints-secret.txt', '/mints/../../cashumints-secret.txt']) {
+    const res = await get(p);
+    assert.equal(res.status, 404, p);
+    assert.doesNotMatch(await res.text(), /secret/, p);
+  }
+  await fs.rm(path.join(root, '..', 'cashumints-secret.txt'), { force: true });
+});
+
+test('dotfiles are refused rather than looked up', async () => {
+  const res = await get('/.env');
+  assert.equal(res.status, 400);
+});
+
+test('every response carries the baseline security headers', async () => {
+  const res = await get('/');
+  assert.equal(res.headers.get('x-content-type-options'), 'nosniff');
+  assert.equal(res.headers.get('referrer-policy'), 'strict-origin-when-cross-origin');
+  assert.equal(res.headers.get('x-frame-options'), 'DENY');
+});
+
+test('safePath refuses what it should and keeps what it should', () => {
+  assert.deepEqual(safePath('/mints/'), ['mints']);
+  assert.deepEqual(safePath('/'), []);
+  assert.equal(safePath('/a\0b'), null);
+  assert.equal(safePath('/.git/config'), null);
+  // Normalised away rather than escaping: the lookup stays inside the root.
+  assert.deepEqual(safePath('/../../etc/passwd'), ['etc', 'passwd']);
+});
+
+test('candidates cover the shapes a static build emits', () => {
+  assert.deepEqual(candidates([]), ['index.html']);
+  assert.deepEqual(candidates(['mints']), ['mints', 'mints/index.html', 'mints.html']);
+});
+
+test('cacheControl is decided by path and extension', () => {
+  assert.match(cacheControl('/mints', '.html'), /must-revalidate/);
+  assert.match(cacheControl('/_astro/x.js', '.js'), /immutable/);
+  assert.match(cacheControl('/robots.txt', '.txt'), /max-age=3600/);
+  assert.match(cacheControl('/favicon.ico', '.ico'), /max-age=604800/);
+});