Your backup job stopped running seven weeks ago. Nothing told you, and nothing will, until the day you need a restore.
Cron has one design flaw and everybody meets it the same way. It says nothing when a job stops working.
Say your backup script exits non-zero. Cron mails the output to the local user’s mailbox. Modern servers have no MTA configured, so that mail goes nowhere. Now say the script never ran at all — someone clobbered the crontab, the server rebooted badly, the binary moved, the disk filled. There is no output to deliver in the first place.
The story ends the same way every time. The job works for months. It stops. You find out weeks later.
So invert the check. Stop waiting for a job to report failure. Expect it to report success, and alert when the report does not arrive. People call this heartbeat monitoring, or a dead man’s switch.
Central Logging’s cron monitoring waits for an HTTP request at the end of a job. Create an API key once, then append a curl to your job:
0 3 * * * /usr/local/bin/backup.sh && curl -fsS "https://logs.example.com/p/YOUR-API-KEY/nightly-backup?state=complete"
The job name (nightly-backup) appears on the fly. You never register it in advance. The && does the real work: the ping fires only when the script exits zero.
Now tell the monitor what to expect. This job reports in daily. A day passes with no ping and you get an alert.
That alert fires whether the script failed, the server died, or someone deleted the crontab. All three look identical from outside, and that is the point.
The curl flags matter more than they look:
-f exits non-zero on HTTP errors, instead of cheerfully succeeding on a 404.-s hides the progress meter, which otherwise becomes cron noise.-S keeps real errors visible despite -s.&& catches non-zero exits. But a failure then produces silence, and you wait out the missed-check-in window before you hear anything. On a daily job that is 24 hours of delay.
Report the failure explicitly and you hear about it now:
#!/bin/bash
# /usr/local/bin/backup.sh
set -euo pipefail
MONITOR="https://logs.example.com/p/YOUR-API-KEY/nightly-backup"
# Report failure on any error, including an unexpected exit.
trap 'curl -fsS "$MONITOR?state=fail&status_code=$?" >/dev/null || true' ERR
pg_dump mydb | gzip > /backups/mydb-$(date +%F).sql.gz
aws s3 sync /backups/ s3://my-backups/
curl -fsS "$MONITOR?state=complete" >/dev/null
set -e plus trap ... ERR makes any failing command report a failure state and stop.
Do not drop set -o pipefail. Without it, gzip succeeding masks pg_dump failing. You then back up an empty file every night while the monitor reports success. That exact failure is common enough to design against.
A failing job is easy. A hanging job hurts. It never reports either way, and each new invocation stacks on the last until the machine falls over.
Send start at the beginning and complete at the end. The monitor learns how long the job normally takes and alerts when a run stays open past that:
#!/bin/bash
set -euo pipefail
MONITOR="https://logs.example.com/p/YOUR-API-KEY/nightly-backup"
trap 'curl -fsS "$MONITOR?state=fail&status_code=$?" >/dev/null || true' ERR
curl -fsS "$MONITOR?state=start" >/dev/null
pg_dump mydb | gzip > /backups/mydb-$(date +%F).sql.gz
aws s3 sync /backups/ s3://my-backups/
curl -fsS "$MONITOR?state=complete" >/dev/null
While you are here, stop overlapping runs at the source with flock:
0 3 * * * /usr/bin/flock -n /tmp/backup.lock /usr/local/bin/backup.sh
-n fails immediately rather than queueing behind a run still in progress. Add the ERR trap and a stuck job now sends you a failure alert on the next scheduled run, instead of a pile of concurrent processes.
The ping takes metadata, which turns the monitor into a small trend view:
START=$(date +%s)
pg_dump mydb | gzip > "$BACKUP_FILE"
DURATION=$(( $(date +%s) - START ))
SIZE=$(stat -c %s "$BACKUP_FILE")
curl -fsS "$MONITOR?state=complete" \
--data-urlencode "message=Backup complete" \
--data-urlencode "metric=duration:$DURATION" \
--data-urlencode "metric=bytes:$SIZE" \
-G >/dev/null
Track the backup size. A backup that drops to a few kilobytes is technically successful and completely worthless. The exit code is zero, the file exists, and the dump is empty because someone rotated the database credentials. Trend the size and that jumps out the same night.
Build the query string with --data-urlencode and -G, not by hand. A message containing spaces or & then cannot corrupt the request.
Sometimes you cannot modify the script. A wrapper handles any command:
#!/bin/bash
# /usr/local/bin/monitored-run
# Usage: monitored-run <job-name> <command> [args...]
set -uo pipefail
API_KEY="YOUR-API-KEY"
BASE="https://logs.example.com/p/$API_KEY"
JOB="$1"; shift
ping() { curl -fsS -G "$BASE/$JOB" --data-urlencode "state=$1" "${@:2}" >/dev/null || true; }
ping start
START=$(date +%s)
OUTPUT=$("$@" 2>&1)
RC=$?
DURATION=$(( $(date +%s) - START ))
if [ $RC -eq 0 ]; then
ping complete --data-urlencode "metric=duration:$DURATION"
else
ping fail \
--data-urlencode "status_code=$RC" \
--data-urlencode "message=$(printf '%s' "$OUTPUT" | tail -c 500)"
fi
# Preserve the original output and exit code for cron's own logging.
printf '%s\n' "$OUTPUT"
exit $RC
0 3 * * * /usr/local/bin/monitored-run nightly-backup /usr/local/bin/backup.sh
15 * * * * /usr/local/bin/monitored-run hourly-sync /usr/local/bin/sync.sh
Sending the last 500 bytes of output on failure is what makes the alert useful. The notification carries the error itself, not the words “job failed”.
Running systemd timers instead of cron? The same approach works through ExecStartPost and ExecStopPost:
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS "https://logs.example.com/p/YOUR-API-KEY/nightly-backup?state=complete"
ExecStartPost runs only when ExecStart succeeded, which gives you the same semantics as &&.
systemd also has OnFailure=, which invokes a unit that reports the failure:
[Unit]
OnFailure=notify-failure@%n.service
Wire up the jobs whose failure is invisible. Roughly in priority order:
certbot renew fails quietly for up to 90 days, then takes the site down.Jobs that fail loudly — anything user-facing — need this less.
Your first alerts will come from cron’s environment, not your code:
PATH is minimal. Cron runs with a stripped PATH, often just /usr/bin:/bin. A script that works in your shell dies under cron with “command not found”. Use absolute paths, or set PATH at the top of the crontab.
% is special. A crontab turns an unescaped % into a newline. date +%F in a crontab line does not do what you meant. Escape it as \%F or move it into a script.
No shell profile. Cron never sources .bashrc or .profile, so the environment variables you rely on are gone. That is why jobs needing credentials work interactively and fail on schedule.
Working directory is $HOME. Relative paths resolve somewhere you did not intend.
All four give you a job that runs, fails instantly, and tells nobody. That is the exact thing heartbeat monitoring catches.
Cron will not tell you when a job stops.
Ping on success. Report failures explicitly with an ERR trap. Send start and complete to catch hangs. Track the backup size as a metric, so an empty backup that “succeeded” cannot hide.
Central Logging puts cron monitoring, log search, alerting, and uptime checks in one binary on your own server. See Cron Monitoring for the full reference, or Deploy to get started.
💌 Get notified on new features and updates