michilisandClaude Opus 5
0ebc8ada54
`Restart=on-failure` with no start limit is an infinite loop by definition: the unit never reaches `failed`, `systemctl status` stays active (auto-restart), and the only evidence is a journal moving at four lines a second. That is how 464 restarts over fifteen hours went unnoticed. All three units now stop after five failures in 120s and run OnFailure=cashumints-alert@%n.service. The window is 120s and not 60s because RestartSec=5s plus a process that takes a few seconds to die can spread five failures past a sixty second window, reset the counter, and loop forever anyway. cashumints-alert@.service is a oneshot that takes the failed unit's name as its instance. Configuration is /etc/cashumints/alert.env: NTFY_URL gets a plain-text body, WEBHOOK_URL gets JSON carrying `content` so one payload fits Discord and Slack-compatible endpoints. With neither set — or the file absent — it still writes to the journal at ERROR via a `<3>` syslog prefix, so `journalctl -p err -t cashumints-alert` is a complete history on a host nobody configured. It cannot become a second thing to debug: each curl is bounded at 10s, each failure falls back to a journal line, and the shell ends in `true`, so the alerter always exits 0. Verified with systemd-analyze verify and by running the ExecStart body against a local sink — the JSON parses, and every branch exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
24 lines
1.1 KiB
Bash
24 lines
1.1 KiB
Bash
# /etc/cashumints/alert.env
|
|
#
|
|
# Read by cashumints-alert@.service, which systemd starts when any of the three units
|
|
# fails. Everything here is optional: with the file absent or both values empty, an
|
|
# alert is still written to the journal at ERROR priority and is readable with
|
|
#
|
|
# journalctl -p err -t cashumints-alert
|
|
#
|
|
# Set one or both to have failures leave the machine.
|
|
#
|
|
# Install it root-owned and not world-readable — a webhook URL is a capability:
|
|
# sudo install -d -m 0755 /etc/cashumints
|
|
# sudo install -m 0640 -o root -g root deploy/alert.env.example /etc/cashumints/alert.env
|
|
# sudo systemctl daemon-reload
|
|
|
|
# An ntfy topic URL. Free and public at ntfy.sh; pick a topic name nobody will guess,
|
|
# because anyone who knows it can read and post to it.
|
|
#NTFY_URL=https://ntfy.sh/cashumints-alerts-CHANGE-ME
|
|
|
|
# Anything that accepts a JSON POST. The body carries `unit`, `host`, `at`, `text` and
|
|
# `content` — the last of which is what Discord and most Slack-compatible endpoints read,
|
|
# so one payload fits all three.
|
|
#WEBHOOK_URL=https://discord.com/api/webhooks/…
|