michilisandClaude Opus 5
0ebc8ada54
`Restart=on-failure` with no start limit is an infinite loop by definition: the unit never reaches `failed`, `systemctl status` stays active (auto-restart), and the only evidence is a journal moving at four lines a second. That is how 464 restarts over fifteen hours went unnoticed. All three units now stop after five failures in 120s and run OnFailure=cashumints-alert@%n.service. The window is 120s and not 60s because RestartSec=5s plus a process that takes a few seconds to die can spread five failures past a sixty second window, reset the counter, and loop forever anyway. cashumints-alert@.service is a oneshot that takes the failed unit's name as its instance. Configuration is /etc/cashumints/alert.env: NTFY_URL gets a plain-text body, WEBHOOK_URL gets JSON carrying `content` so one payload fits Discord and Slack-compatible endpoints. With neither set — or the file absent — it still writes to the journal at ERROR via a `<3>` syslog prefix, so `journalctl -p err -t cashumints-alert` is a complete history on a host nobody configured. It cannot become a second thing to debug: each curl is bounded at 10s, each failure falls back to a journal line, and the shell ends in `true`, so the alerter always exits 0. Verified with systemd-analyze verify and by running the ExecStart body against a local sink — the JSON parses, and every branch exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
96 lines
4.7 KiB
Desktop File
96 lines
4.7 KiB
Desktop File
# /etc/systemd/system/cashumints-web.service
|
|
#
|
|
# The frontend is static: `output: 'static'` in astro.config.mjs, and the whole site is
|
|
# produced ahead of time. There is no frontend build to keep alive, so this unit is a
|
|
# build rather than a daemon — one shot of `pnpm build`, which compiles shared/, renders
|
|
# a social card per mint and prerenders every page from the live API. The daemon that
|
|
# hands the result out is cashumints-site.service.
|
|
#
|
|
# Run it after a deploy:
|
|
# sudo systemctl start cashumints-web
|
|
#
|
|
# Mint pages are prerendered, so new mints and new review counts only appear at the
|
|
# next build; pair this with a .timer for the nightly rebuild. There is deliberately no
|
|
# [Install] section — a rebuild should be scheduled, not fired on every boot.
|
|
|
|
[Unit]
|
|
Description=Rebuild the cashumints.space static site
|
|
# Every page's data comes from the API over loopback, so the API has to be up.
|
|
# Requires= rather than Wants=: a dead API should abort the build, not replace a good
|
|
# site with an empty one.
|
|
Requires=cashumints.service
|
|
After=cashumints.service network-online.target
|
|
Wants=network-online.target
|
|
|
|
# Carry a failure off the machine. `%n` is this unit's own name, so the alert says
|
|
# which one died. cashumints-alert@.service writes to the journal at ERROR always and
|
|
# curls NTFY_URL or WEBHOOK_URL from /etc/cashumints/alert.env when either is set.
|
|
OnFailure=cashumints-alert@%n.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
User=cashumints
|
|
Group=cashumints
|
|
WorkingDirectory=/home/cashumints/CashuMints.space
|
|
|
|
# Where the published copy lands. Shared with the API and the site server, and created
|
|
# by systemd with this unit's ownership if it is not there yet.
|
|
StateDirectory=cashumints
|
|
|
|
Environment=NODE_ENV=production
|
|
# Where the build reaches the API. Must match PORT= in cashumints.service.
|
|
Environment=API_URL=http://127.0.0.1:8788
|
|
Environment=SITE_URL=https://cashumints.space
|
|
# Browser-facing origin. Empty means same origin: islands fetch /api/... and nginx
|
|
# forwards it. Set this only if the API ever moves to its own hostname. Declared here
|
|
# even though it is empty, because systemd's environment wins over .env — so what a
|
|
# production build emits cannot drift with an edit to that file.
|
|
Environment=PUBLIC_API_URL=
|
|
|
|
# After= orders the start; it does not wait for the port to accept connections. At boot
|
|
# the API is still opening its database and probing, so block until it reports healthy
|
|
# rather than letting the first fetch die on ECONNREFUSED. /api/health answers 503 until
|
|
# it is genuinely ready, and curl -f treats that as a failure, so the loop keeps waiting.
|
|
ExecStartPre=/usr/bin/timeout 90 /bin/sh -c 'until curl -sf -o /dev/null http://127.0.0.1:8788/api/health; do sleep 1; done'
|
|
# Check `which pnpm` on the host: a corepack or pnpm-home install sits outside /usr/bin,
|
|
# and systemd's PATH does not include it.
|
|
ExecStart=/usr/bin/pnpm build
|
|
|
|
# Publish, as a separate step from building.
|
|
#
|
|
# `astro build` empties dist before it writes, so the site server cannot read dist
|
|
# directly — a nightly rebuild would be a nightly minute of 404s. It serves this copy
|
|
# instead, and the copy is only touched once a build has succeeded: a failed build
|
|
# leaves the previous site up rather than replacing it with a half-written one, which is
|
|
# the same reason Requires=cashumints.service is above.
|
|
#
|
|
# --delay-updates stages the changed files and renames them in at the end, so the window
|
|
# where the tree is a mix of two builds is a rename rather than a whole transfer, and
|
|
# --delete-after keeps removals from landing before their replacements. Unchanged files
|
|
# — every hashed asset and card, which is nearly all of it — are not touched at all.
|
|
ExecStartPost=/usr/bin/rsync -a --delete-after --delay-updates web/dist/ /var/lib/cashumints/web/
|
|
|
|
# ~200 prerendered pages plus a card per mint. Minutes, not seconds, on a small VPS, and
|
|
# TimeoutStartSec is what bounds a Type=oneshot.
|
|
TimeoutStartSec=1800
|
|
# A nightly rebuild should not starve the API it is reading from.
|
|
Nice=10
|
|
|
|
# The site server runs as cashumints and reads its own files, so this no longer has to
|
|
# be world-readable — it was 0022 for nginx, back when nginx opened the files as
|
|
# www-data. Kept at 0022 anyway: rsync preserves these modes into the published copy,
|
|
# and a readable static site is easier to inspect than one that needs sudo.
|
|
UMask=0022
|
|
|
|
NoNewPrivileges=true
|
|
PrivateTmp=true
|
|
PrivateDevices=true
|
|
# ProtectHome is deliberately absent, unlike in cashumints.service: this unit writes
|
|
# inside /home/cashumints — web/dist, web/public/og, web/src/generated and the pnpm
|
|
# store are all under it.
|
|
ProtectSystem=full
|
|
ProtectKernelTunables=true
|
|
ProtectKernelModules=true
|
|
ProtectControlGroups=true
|
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|