Cron mails your job’s output to a mailbox nobody reads.
That is the whole problem. Your backup script printed the error. Cron dutifully collected it, handed it to the local mail transfer agent, and on most modern servers there is no local mail transfer agent. The output went nowhere.
So when the job breaks, you go looking. /var/log/syslog has a line saying cron ran something. It does not have what that something said.
This guide covers where cron output actually goes, and how to put it somewhere you can search.
Cron captures everything a job writes to stdout and stderr. Then it does one of three things.
With MAILTO set and a working MTA, it mails you. This is the design. It assumed a machine where mail worked and someone read root’s mailbox daily.
With no MTA, delivery fails and the output is discarded. This is most servers now. Cron logs a delivery failure at best.
With MAILTO="", cron discards the output on purpose. Plenty of crontabs have this line, added years ago by someone tired of the noise.
Check yours:
crontab -l | head -5
grep -rn MAILTO /etc/crontab /etc/cron.d/ 2>/dev/null
This trips people up, so look at it directly:
grep CRON /var/log/syslog | tail -5
# Debian/Ubuntu; on RHEL family it is /var/log/cron
Or through journald:
journalctl -u cron --since "1 hour ago"
journalctl _COMM=cron --since today
You get lines like:
Aug 7 03:00:01 web01 CRON[12345]: (root) CMD (/usr/local/bin/backup.sh)
That says cron started the command. It does not say what the command printed, how long it took, or what it exited with. Everything you actually want to know is in the output cron just threw away.
The two-second fix:
0 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1
2>&1 matters and goes last. It sends stderr wherever stdout is already going, and errors are the reason you are here.
This works. It also gives you a file per job, on the machine where the job ran, growing forever until you write a logrotate rule. Three servers and five jobs is fifteen files and fifteen SSH sessions. It solves the wrong half of the problem.
loggerlogger writes to syslog, which means journald picks it up and your existing log shipper carries it away:
0 3 * * * /usr/local/bin/backup.sh 2>&1 | logger -t backup
Now journalctl -t backup works, and if you already ship journald with clagent the output lands centrally with no extra plumbing. See sending journalctl logs if you have not set that up.
This is a good default. It has one flaw, and it is the same flaw as the next option, so read on before you commit to it.
No agent needed. Central Logging accepts a plain HTTP POST:
/usr/local/bin/backup.sh 2>&1 | \
curl -fsS --data-binary @- "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
ingest_logs takes newline-separated lines and stores one row per line. Which brings us to the flaw.
Your job fails and prints this:
line1
Traceback (most recent call last):
File "x.py", line 3
ValueError: boom
Pipe that in and you get four rows. Four separate log entries, with no field tying them together, and no indication which job produced them. Search for ValueError and you find one line with no context. The traceback that explains it is a different row.
Worse, none of those rows are JSON, so they are invisible to any query using json_extract.
The fix is to send one row per run, with the output inside a field.
#!/bin/sh
# /usr/local/bin/cronlog
# Usage: cronlog <job-name> <command> [args...]
set -u
INGEST="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
JOB="$1"; shift
OUT=$(mktemp)
START=$(date +%s)
"$@" > "$OUT" 2>&1
RC=$?
DURATION=$(( $(date +%s) - START ))
LEVEL=info
[ "$RC" -eq 0 ] || LEVEL=error
jq -Rs --arg job "$JOB" --arg level "$LEVEL" \
--argjson rc "$RC" --argjson secs "$DURATION" \
'{job: $job, level: $level, status_code: $rc, duration_s: $secs, output: .}' \
-c < "$OUT" | curl -fsS --data-binary @- "$INGEST" >/dev/null || true
rm -f "$OUT"
exit $RC
0 3 * * * /usr/local/bin/cronlog nightly-backup /usr/local/bin/backup.sh
15 * * * * /usr/local/bin/cronlog hourly-sync /usr/local/bin/sync.sh
Four details in there are deliberate.
jq -Rs reads the whole file as one JSON string. -R treats input as raw text instead of JSON, -s slurps it into a single value. Newlines become \n inside the string, so a fifteen-line stack trace stays in one row and stays searchable.
-c keeps the JSON on one line. The endpoint splits on newlines. Pretty-printed JSON would become one row per line of JSON, which is the bug you just fixed.
|| true after curl. The ping must never fail the job. A log server that is down should not turn a working backup into a failing one.
exit $RC at the end. The wrapper reports the original exit code, so && chains and MAILTO behave exactly as before.
No jq on the box? Any language you already have will do. The requirement is one line of JSON with the output as a string value.
Every run is now one row with fields. See querying logs with SQL for the full syntax.
Which jobs failed today?
SELECT datetime("timestamp", 'unixepoch') AS at,
json_extract(msg, '$.job') AS job,
json_extract(msg, '$.status_code') AS exit_code,
json_extract(msg, '$.output') AS output
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.status_code') != 0
AND "timestamp" > strftime('%s', 'now', 'start of day')
ORDER BY "timestamp" DESC;
Is a job getting slower?
SELECT date("timestamp", 'unixepoch') AS day,
max(json_extract(msg, '$.duration_s')) AS slowest_run_s
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.job') = 'nightly-backup'
GROUP BY day
ORDER BY day DESC
LIMIT 30;
A backup that has drifted from four minutes to fifty is a problem you want to see in that column, not in a timeout at 3am.
Did a job run at all? You cannot answer that from its output. A job that never started prints nothing, and nothing is what a healthy silent job also prints.
This is the honest limit of everything above.
Shipping output tells you what a job said. It cannot tell you that a job said nothing, because a missing log line looks exactly like a quiet success. The crontab someone deleted in March produces no error to ship.
For that you need the job to report in, and something to notice when the report does not arrive. Add a ping alongside the wrapper:
MONITOR="https://logs.example.com/p/YOUR-API-KEY/$JOB"
if [ "$RC" -eq 0 ]; then
curl -fsS "$MONITOR?state=complete" >/dev/null || true
else
curl -fsS -G "$MONITOR" --data-urlencode "state=fail" \
--data-urlencode "status_code=$RC" >/dev/null || true
fi
The two work together. The ping tells you a job broke or vanished. The shipped output tells you why, without an SSH session.
How to monitor cron jobs and get alerted when they fail covers the monitoring half properly, including hung jobs and systemd timers.
Do not ship secrets. set -x in a script prints every command with its arguments expanded, database URLs and API keys included. Anything your job prints, the wrapper stores.
Cap the output. A job that loops printing errors can produce megabytes. Truncate before sending:
tail -c 10000 "$OUT" | jq -Rs ...
The last ten kilobytes hold the error. The first ten megabytes hold the run-up to it.
Start with option 2 if you already ship journald. Move to the wrapper the first time a stack trace arrives as fifteen unrelated rows.
💌 Get notified on new features and updates