fix: harden deploys and close top security holes after /events outage

Isolate next dev from production .next, add build-guard/atomic deploy/health watchdog, error boundaries, and fix JWT startup, meetup leaks, media path traversal, SVG/memory uploads, and JSON-LD escaping. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-18 19:14:57 +02:00
parent 495289232b
commit ede8817583
22 changed files with 700 additions and 86 deletions
@@ -0,0 +1,85 @@
+# Ops: keeping the site up
+
+These files harden the deploy/run path so a stray `next dev` or a half-finished
+build can no longer take the public site down (root cause of the `/events` 500).
+
+## What's here
+
+| Path | Purpose |
+|------|---------|
+| `frontend/scripts/verify-prod-build.sh` | Fails fast if `.next` is missing or dev-contaminated. Wired as `ExecStartPre`. |
+| `scripts/deploy-frontend.sh` | Atomic build-verify-swap-restart deploy. Use this to deploy the frontend. |
+| `scripts/healthcheck.sh` | Watchdog: probes `/`, `/events`, `/blog`, `/api/health`; restarts frontend after repeated failures. |
+| `ops/systemd/bbe-frontend.service.d/verify-build.conf` | Drop-in adding the build guard to `bbe-frontend.service`. |
+| `ops/systemd/bbe-site-healthcheck.{service,timer}` | Runs the watchdog every 5 minutes. |
+
+## Layered defenses
+
+1. **Isolation** — `next dev` now writes to `.next-dev` (`NEXT_DIST_DIR` in
+   `frontend/package.json` + `distDir` in `frontend/next.config.js`), so it can
+   never overwrite the production `.next`.
+2. **Fail-fast boot** — the `ExecStartPre` guard refuses to start on a bad build.
+3. **Atomic deploys** — `deploy-frontend.sh` builds into `.next-build`, verifies,
+   then renames into place; the running server never sees a partial `.next`.
+4. **Self-healing** — the watchdog restarts a wedged frontend within minutes.
+5. **Graceful UI** — `app/error.tsx` / `app/global-error.tsx` render a retry page
+   instead of a bare 500 if a request-time route ever throws.
+
+## Install (requires sudo)
+
+One-shot installer (preferred):
+
+```bash
+cd /home/bbe/BelgianBitcoinEmbassy
+sudo ops/install-systemd.sh
+sudo systemctl restart bbe-backend   # after backend code changes
+```
+
+Or manually:
+
+```bash
+cd /home/bbe/BelgianBitcoinEmbassy
+
+# 1. Build guard (ExecStartPre)
+sudo mkdir -p /etc/systemd/system/bbe-frontend.service.d
+sudo cp ops/systemd/bbe-frontend.service.d/verify-build.conf \
+        /etc/systemd/system/bbe-frontend.service.d/verify-build.conf
+
+# 2. Health watchdog
+sudo cp ops/systemd/bbe-site-healthcheck.service /etc/systemd/system/
+sudo cp ops/systemd/bbe-site-healthcheck.timer   /etc/systemd/system/
+
+sudo systemctl daemon-reload
+sudo systemctl restart bbe-frontend
+sudo systemctl enable --now bbe-site-healthcheck.timer
+```
+
+Verify:
+
+```bash
+systemctl status bbe-frontend --no-pager
+systemctl list-timers bbe-site-healthcheck --no-pager
+journalctl -u bbe-healthcheck -n 20 --no-pager
+```
+
+## Deploying the frontend from now on
+
+Do **not** run `npm run build` directly in the live tree and then restart. Use:
+
+```bash
+scripts/deploy-frontend.sh
+```
+
+It refuses to build when free memory is low (`MIN_FREE_MB`, default 600), builds
+into `.next-build`, verifies, atomically swaps to `.next`, and restarts the
+service. The previous build is kept at `.next-prev` for a quick manual rollback.
+
+## Note on the watchdog restarting the service
+
+`healthcheck.sh` runs `systemctl restart bbe-frontend`. The systemd service above
+runs as root, so this works out of the box. If you ever run the script as the
+`bbe` user instead, grant a narrow sudoers rule:
+
+```
+bbe ALL=(root) NOPASSWD: /usr/bin/systemctl restart bbe-frontend
+```