How to show only errors in journalctl

Introduction

journalctl -p err -b

That is the command. The rest of this guide is about the times it returns nothing and the journal is visibly full of errors, because that happens more often than the command failing.

The eight levels

journalctl uses the syslog priorities. Lower numbers are more severe.

You can use either the name or the number. -p err and -p 3 are the same command.

A single level means that level and everything worse

This is the part people misread. -p err does not show errors only. It shows priority 3 and lower numbers — err, crit, alert and emerg.

journalctl -p err        # 0, 1, 2 and 3
journalctl -p warning    # 0 through 4
journalctl -p debug      # everything

That is almost always what you want. A single priority acts as a ceiling on the number, which is a floor on the severity.

Getting exactly one level

Pass a range in the form FROM..TO. Both ends are included.

journalctl -p 3..3       # errors only, nothing more severe
journalctl -p 0..3       # the same set as -p err, written out
journalctl -p 4..6       # warning, notice and info, no errors

-p 3..3 is the one worth remembering. It answers “what errors did we log”, without the kernel’s panic-adjacent messages on top.

Narrowing it further

Priority is one match among several, so it ANDs with everything else:

journalctl -p err -u nginx                    # one unit
journalctl -p err -b                          # this boot
journalctl -p err -b -1                       # the boot that ended badly
journalctl -p err --since "-1h"               # the last hour
journalctl -p err --since today -o short-iso  # today, with sortable timestamps

Two that are worth having in your shell history:

journalctl -p err -b --no-pager | tail -50
journalctl -p err -f

The first is the summary you want after a reboot. The second follows new errors as they arrive — see following logs in real time with journalctl -f for what -f does and does not guarantee.

Why -p err finds nothing in an application full of errors

Here is the thing that costs people an afternoon.

The priority is set by whatever sent the message, not by what the message says. journalctl filters on a PRIORITY field. It does not read the text. A line saying ERROR: payment gateway timeout carries whatever priority its sender attached, and for most applications that is not err.

When a service writes to stdout or stderr, systemd captures the stream and tags every line with the unit’s SyslogLevel=. That setting defaults to info, and it applies to stderr exactly as it applies to stdout. So a program that dutifully writes its errors to stderr still lands in the journal at priority 6, and journalctl -p err -u yourapp returns nothing.

Confirm it by looking at what priorities the unit actually emits:

journalctl -u yourapp -o json --output-fields=PRIORITY,MESSAGE -n 20

If every line says "PRIORITY" : "6", this is your problem.

Three ways to fix it

Search the text instead. The fastest answer, and it works today:

journalctl -u yourapp -g "error|fatal|panic" --since "-1h"

-g takes a regular expression and matches against the message text. See searching journalctl logs for a specific string for the details, including a case-sensitivity rule that catches people out.

Raise the level for the whole unit. If a service only ever writes things worth noticing, tag the lot:

[Service]
SyslogLevel=warning

Blunt, but honest — it stops the unit from claiming that everything is fine.

Prefix the lines at the source. systemd reads a <N> prefix on each line and uses it as that line’s priority, stripping the prefix before it stores the message. The behaviour is controlled by SyslogLevelPrefix=, which defaults to true, so it works without configuring anything:

echo "<3>database connection refused"   # arrives as priority 3, err
echo "<6>listening on :8080"            # arrives as priority 6, info

Most logging libraries can be told to emit that prefix. It is the only one of the three that gets the priorities genuinely right, because the application is the only thing that knows which lines are errors.

The second reason: not every error is a log line

-p err reads the journal. Some failures never reach it.

A unit that fails to start writes little or nothing itself. The failure lives in systemd’s own view:

systemctl --failed
systemctl status yourapp

And a process killed by the OOM killer leaves its evidence in the kernel ring buffer, at priority 2 rather than 3:

journalctl -k -p err -b
journalctl -k -g "Out of memory" -b

Worth checking both before concluding that nothing went wrong.

Errors across every host at once

The commands above answer “what went wrong on this machine”. During an incident that is the wrong question. You want the errors from all of them, in one list, sorted by time.

Ten hosts is ten SSH sessions and ten separate answers, and you have to hold the ordering in your head. It is also the wrong tool for anything recurring: to notice an error you have to already be looking.

Shipping the journals somewhere central fixes both. The CL Agent reads journald and forwards it, keeping the fields intact, so PRIORITY and _SYSTEMD_UNIT are still there on the other side.

Querying errors once the logs have landed

Central Logging stores each line in SQLite. -p err becomes a WHERE clause:

SELECT datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000, 'unixepoch') AS at,
       json_extract(msg, '$._HOSTNAME')     AS host,
       json_extract(msg, '$._SYSTEMD_UNIT') AS unit,
       json_extract(msg, '$.MESSAGE')       AS message
FROM logs
WHERE json_valid(msg)
  AND CAST(json_extract(msg, '$.PRIORITY') AS INTEGER) <= 3
ORDER BY at DESC
LIMIT 50;

Two details in there are load-bearing.

CAST is not optional. journald stores PRIORITY as a string, so json_extract(msg, '$.PRIORITY') <= 3 compares text against a number. SQLite sorts every text value above every integer, so that test matches zero rows and raises no error. You get an empty result and no hint why.

The same rule points the other way for thresholds. Without a cast, a >= comparison matches every text value rather than none — a disk-usage rule written as json_extract(msg, '$.used_pct') >= 85 fires on a disk at 3% as soon as one producer quotes the number. >= over-matches, <= under-matches, and neither says a word.

Keep the json_valid(msg) guard. json_extract errors on the first non-JSON line it meets. The guard in WHERE protects the terms after it, because SQLite short-circuits AND. It does not protect the SELECT list, so the guard has to remove the bad rows rather than merely precede them.

Catching the errors that PRIORITY misses

The query above has exactly the blind spot the command line has: an application logging errors at priority 6 will not appear. Search the text as well, using the full-text index:

SELECT datetime(logs.timestamp, 'unixepoch') AS received, logs.msg
FROM logs_fts
JOIN logs ON logs.id = logs_fts.rowid
WHERE logs_fts MATCH 'error OR fatal OR panic'
ORDER BY rank
LIMIT 50;

In practice you want both. Priority catches what the system considers serious; text catches what your application considers serious. Neither is a superset of the other.

Alerting instead of looking

Once the query is right, the alert rule is the same query with the columns thrown away. Central Logging runs each rule and only checks whether the result set had anything in it, so SELECT 1 ... LIMIT 1 is the whole job:

SELECT 1 FROM logs
WHERE json_valid(msg)
  AND CAST(json_extract(msg, '$.PRIORITY') AS INTEGER) <= 3
  AND timestamp > CAST(strftime('%s', 'now', '-10 minutes') AS INTEGER)
LIMIT 1;

Set it to alert on results. Rules are evaluated every five minutes and the notification goes out within a minute of firing, so pick a window a little wider than the interval — ten minutes here — or an error landing between two evaluations slips through the gap.

The text-matching counterpart, for the applications that log at info:

SELECT 1 FROM logs_fts WHERE msg MATCH 'panic OR fatal' LIMIT 1;

Two things worth knowing before you rely on either. There is no all-clear message — a rule that stops matching simply stops firing. And the Max Frequency setting on the rule is what stops a broken deploy from sending you four hundred notifications; the shortest option is five minutes, matching the evaluation interval.

Alerting on errors in your logs covers the rule shapes in more depth, including the one almost nobody writes: alert when a source goes quiet. An application that has crashed logs no errors at all, and every rule on this page stays silent for it.

Where to go next

• How to search journalctl logs for a specific string — -g, and finding the errors that -p cannot see.
• How to view journalctl logs for a specific time range — --since and --until, and the midnight trap.
• How to view logs for a systemd service — -u in depth, and what to check when a unit prints nothing.
• How to find out why a Linux server rebooted — -b -1 and reading the errors from the boot that ended.
• How to get alerted on errors in your logs — rules that fire, and the one that fires when nothing arrives.
• Alerting — the reference for rules, channels and frequency.