`Restart=on-failure` with no start limit is an infinite loop by definition: the
unit never reaches `failed`, `systemctl status` stays active (auto-restart), and
the only evidence is a journal moving at four lines a second. That is how 464
restarts over fifteen hours went unnoticed.
All three units now stop after five failures in 120s and run
OnFailure=cashumints-alert@%n.service. The window is 120s and not 60s because
RestartSec=5s plus a process that takes a few seconds to die can spread five
failures past a sixty second window, reset the counter, and loop forever anyway.
cashumints-alert@.service is a oneshot that takes the failed unit's name as its
instance. Configuration is /etc/cashumints/alert.env: NTFY_URL gets a plain-text
body, WEBHOOK_URL gets JSON carrying `content` so one payload fits Discord and
Slack-compatible endpoints. With neither set — or the file absent — it still
writes to the journal at ERROR via a `<3>` syslog prefix, so `journalctl -p err -t
cashumints-alert` is a complete history on a host nobody configured.
It cannot become a second thing to debug: each curl is bounded at 10s, each
failure falls back to a journal line, and the shell ends in `true`, so the alerter
always exits 0. Verified with systemd-analyze verify and by running the ExecStart
body against a local sink — the JSON parses, and every branch exits 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The unit's ExecStart named src/index.ts, so every start depended on the host
having Node 22.18 or newer for native type stripping. A deploy onto a host with
Node 20 met ERR_UNKNOWN_FILE_EXTENSION, exited in under a second, and was
restarted 464 times over fifteen hours with nothing anywhere going red.
api/tsconfig.json now emits to api/dist. The source keeps its explicit .ts import
specifiers, which is what makes `node --watch src/index.ts` work in development;
rewriteRelativeImportExtensions turns them into .js on the way out, so what runs
in production is ordinary ESM that any Node from 20.18 up will start.
`pnpm build` builds shared, then api, then web. `pnpm dev` is unchanged.
deploy/ is tracked rather than ignored: the unit files are the thing an operator
copies to /etc/systemd/system, and the alert unit added next has to live
somewhere a deploy can find it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
