michilisandClaude Opus 5
0ebc8ada54
`Restart=on-failure` with no start limit is an infinite loop by definition: the unit never reaches `failed`, `systemctl status` stays active (auto-restart), and the only evidence is a journal moving at four lines a second. That is how 464 restarts over fifteen hours went unnoticed. All three units now stop after five failures in 120s and run OnFailure=cashumints-alert@%n.service. The window is 120s and not 60s because RestartSec=5s plus a process that takes a few seconds to die can spread five failures past a sixty second window, reset the counter, and loop forever anyway. cashumints-alert@.service is a oneshot that takes the failed unit's name as its instance. Configuration is /etc/cashumints/alert.env: NTFY_URL gets a plain-text body, WEBHOOK_URL gets JSON carrying `content` so one payload fits Discord and Slack-compatible endpoints. With neither set — or the file absent — it still writes to the journal at ERROR via a `<3>` syslog prefix, so `journalctl -p err -t cashumints-alert` is a complete history on a host nobody configured. It cannot become a second thing to debug: each curl is bounded at 10s, each failure falls back to a journal line, and the shell ends in `true`, so the alerter always exits 0. Verified with systemd-analyze verify and by running the ExecStart body against a local sink — the JSON parses, and every branch exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
71 lines
2.7 KiB
Desktop File
71 lines
2.7 KiB
Desktop File
# /etc/systemd/system/cashumints.service
|
|
[Unit]
|
|
Description=cashumints.space indexer and API
|
|
Wants=network-online.target
|
|
After=network-online.target
|
|
# Stop after five failures in two minutes rather than restarting forever.
|
|
#
|
|
# The window is 120s and not 60s because RestartSec=5s below means five attempts take
|
|
# a little over twenty seconds of restarts plus however long each attempt lives before
|
|
# it dies. A process that fails *slowly* — a database that times out, a port that takes
|
|
# four seconds to refuse — can spread five failures past a sixty second window and reset
|
|
# the counter forever, which is the loop this is supposed to stop. 120s covers that.
|
|
#
|
|
# The failure this exists for: ExecStart named a .ts file, /usr/bin/node was 20, and
|
|
# every start died in under a second. 464 restarts over fifteen hours, and because
|
|
# Restart=on-failure without a start limit never reaches a `failed` state, nothing
|
|
# anywhere went red. Both keys belong to [Unit] — under [Service] systemd only warns and
|
|
# ignores them.
|
|
StartLimitIntervalSec=120
|
|
StartLimitBurst=5
|
|
|
|
# Carry a failure off the machine. `%n` is this unit's own name, so the alert says
|
|
# which one died. cashumints-alert@.service writes to the journal at ERROR always and
|
|
# curls NTFY_URL or WEBHOOK_URL from /etc/cashumints/alert.env when either is set.
|
|
OnFailure=cashumints-alert@%n.service
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=cashumints
|
|
Group=cashumints
|
|
WorkingDirectory=/home/cashumints/CashuMints.space/api
|
|
|
|
# StateDirectory creates /var/lib/cashumints with the service user's ownership.
|
|
StateDirectory=cashumints
|
|
Environment=NODE_ENV=production
|
|
Environment=PORT=8788
|
|
Environment=DB_PATH=/var/lib/cashumints/cashumints.db
|
|
Environment=ICON_DIR=/var/lib/cashumints/icons
|
|
|
|
# Compiled JavaScript, run by the distribution's own node.
|
|
#
|
|
# This line used to read `src/index.ts`, which made every start depend on the host
|
|
# having Node 22.18 or newer for native type stripping. A host with Node 20 answered
|
|
# that with ERR_UNKNOWN_FILE_EXTENSION in under a second, 464 times over fifteen hours,
|
|
# and nothing anywhere went red. `pnpm build` now emits api/dist, so what runs here is
|
|
# ordinary ESM and any Node from 20.18 up will start it.
|
|
#
|
|
# Deliberately /usr/bin/node and nothing else: an nvm or fnm path is invisible to this
|
|
# unit's ProtectHome and breaks silently at the next version bump.
|
|
ExecStart=/usr/bin/node --env-file-if-exists=../.env dist/index.js
|
|
|
|
Restart=on-failure
|
|
RestartSec=5s
|
|
KillSignal=SIGTERM
|
|
TimeoutStopSec=30s
|
|
UMask=0027
|
|
|
|
NoNewPrivileges=true
|
|
PrivateTmp=true
|
|
ProtectSystem=strict
|
|
ProtectHome=read-only
|
|
ReadWritePaths=/var/lib/cashumints
|
|
PrivateDevices=true
|
|
ProtectKernelTunables=true
|
|
ProtectKernelModules=true
|
|
ProtectControlGroups=true
|
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|