# Ops: keeping the site up

These files harden the deploy/run path so a stray `next dev` or a half-finished
build can no longer take the public site down (root cause of the `/events` 500).

## What's here

| Path | Purpose |
|------|---------|
| `frontend/scripts/verify-prod-build.sh` | Fails fast if `.next` is missing or dev-contaminated. Wired as `ExecStartPre`. |
| `scripts/deploy-frontend.sh` | Atomic build-verify-swap-restart deploy. Use this to deploy the frontend. |
| `scripts/healthcheck.sh` | Watchdog: probes `/`, `/events`, `/blog`, `/api/health`; restarts frontend after repeated failures. |
| `ops/systemd/bbe-frontend.service.d/verify-build.conf` | Drop-in adding the build guard to `bbe-frontend.service`. |
| `ops/systemd/bbe-site-healthcheck.{service,timer}` | Runs the watchdog every 5 minutes. |

## Layered defenses

1. **Isolation** — `next dev` now writes to `.next-dev` (`NEXT_DIST_DIR` in
   `frontend/package.json` + `distDir` in `frontend/next.config.js`), so it can
   never overwrite the production `.next`.
2. **Fail-fast boot** — the `ExecStartPre` guard refuses to start on a bad build.
3. **Atomic deploys** — `deploy-frontend.sh` builds into `.next-build`, verifies,
   then renames into place; the running server never sees a partial `.next`.
4. **Self-healing** — the watchdog restarts a wedged frontend within minutes.
5. **Graceful UI** — `app/error.tsx` / `app/global-error.tsx` render a retry page
   instead of a bare 500 if a request-time route ever throws.

## Install (requires sudo)

One-shot installer (preferred):

```bash
cd /home/bbe/BelgianBitcoinEmbassy
sudo ops/install-systemd.sh
sudo systemctl restart bbe-backend   # after backend code changes
```

Or manually:

```bash
cd /home/bbe/BelgianBitcoinEmbassy

# 1. Build guard (ExecStartPre)
sudo mkdir -p /etc/systemd/system/bbe-frontend.service.d
sudo cp ops/systemd/bbe-frontend.service.d/verify-build.conf \
        /etc/systemd/system/bbe-frontend.service.d/verify-build.conf

# 2. Health watchdog
sudo cp ops/systemd/bbe-site-healthcheck.service /etc/systemd/system/
sudo cp ops/systemd/bbe-site-healthcheck.timer   /etc/systemd/system/

sudo systemctl daemon-reload
sudo systemctl restart bbe-frontend
sudo systemctl enable --now bbe-site-healthcheck.timer
```

Verify:

```bash
systemctl status bbe-frontend --no-pager
systemctl list-timers bbe-site-healthcheck --no-pager
journalctl -u bbe-healthcheck -n 20 --no-pager
```

## Deploying the frontend from now on

Do **not** run `npm run build` directly in the live tree and then restart. Use:

```bash
scripts/deploy-frontend.sh
```

It refuses to build when free memory is low (`MIN_FREE_MB`, default 600), builds
into `.next-build`, verifies, atomically swaps to `.next`, and restarts the
service. The previous build is kept at `.next-prev` for a quick manual rollback.

## Note on the watchdog restarting the service

`healthcheck.sh` runs `systemctl restart bbe-frontend`. The systemd service above
runs as root, so this works out of the box. If you ever run the script as the
`bbe` user instead, grant a narrow sudoers rule:

```
bbe ALL=(root) NOPASSWD: /usr/bin/systemctl restart bbe-frontend
```