BelgianBitcoinEmbassy/ops/README.md

Ops: keeping the site up

These files harden the deploy/run path so a stray next dev or a half-finished build can no longer take the public site down (root cause of the /events 500).

What's here

Path	Purpose
frontend/scripts/verify-prod-build.sh	Fails fast if .next is missing or dev-contaminated. Wired as ExecStartPre.
scripts/deploy-frontend.sh	Atomic build-verify-swap-restart deploy. Use this to deploy the frontend.
scripts/healthcheck.sh	Watchdog: probes /, /events, /blog, /api/health; restarts frontend after repeated failures.
ops/systemd/bbe-frontend.service.d/verify-build.conf	Drop-in adding the build guard to bbe-frontend.service.
ops/systemd/bbe-site-healthcheck.{service,timer}	Runs the watchdog every 5 minutes.

Layered defenses

1. Isolation — next dev now writes to .next-dev (NEXT_DIST_DIR in frontend/package.json + distDir in frontend/next.config.js), so it can never overwrite the production .next.
2. Fail-fast boot — the ExecStartPre guard refuses to start on a bad build.
3. Atomic deploys — deploy-frontend.sh builds into .next-build, verifies, then renames into place; the running server never sees a partial .next.
4. Self-healing — the watchdog restarts a wedged frontend within minutes.
5. Graceful UI — app/error.tsx / app/global-error.tsx render a retry page instead of a bare 500 if a request-time route ever throws.

Install (requires sudo)

One-shot installer (preferred):

cd /home/bbe/BelgianBitcoinEmbassy
sudo ops/install-systemd.sh
sudo systemctl restart bbe-backend   # after backend code changes

Or manually:

cd /home/bbe/BelgianBitcoinEmbassy

# 1. Build guard (ExecStartPre)
sudo mkdir -p /etc/systemd/system/bbe-frontend.service.d
sudo cp ops/systemd/bbe-frontend.service.d/verify-build.conf \
        /etc/systemd/system/bbe-frontend.service.d/verify-build.conf

# 2. Health watchdog
sudo cp ops/systemd/bbe-site-healthcheck.service /etc/systemd/system/
sudo cp ops/systemd/bbe-site-healthcheck.timer   /etc/systemd/system/

sudo systemctl daemon-reload
sudo systemctl restart bbe-frontend
sudo systemctl enable --now bbe-site-healthcheck.timer

Verify:

systemctl status bbe-frontend --no-pager
systemctl list-timers bbe-site-healthcheck --no-pager
journalctl -u bbe-healthcheck -n 20 --no-pager

Deploying the frontend from now on

Do not run npm run build directly in the live tree and then restart. Use:

scripts/deploy-frontend.sh

It refuses to build when free memory is low (MIN_FREE_MB, default 600), builds into .next-build, verifies, atomically swaps to .next, and restarts the service. The previous build is kept at .next-prev for a quick manual rollback.

Note on the watchdog restarting the service

healthcheck.sh runs systemctl restart bbe-frontend. The systemd service above runs as root, so this works out of the box. If you ever run the script as the bbe user instead, grant a narrow sudoers rule:

bbe ALL=(root) NOPASSWD: /usr/bin/systemctl restart bbe-frontend