How to follow logs in real time with journalctl -f

Introduction

journalctl -f shows you one machine.

That is fine when you know which machine broke. Most of the time you are watching the wrong one. This guide covers following the journal live, the flags that make it readable, the two things that make it look broken, and what to do when the interesting log line is on a server you are not looking at.

The short answer

journalctl -f

-f prints the last few entries and then waits, printing each new one as it arrives. Ctrl-C stops it.

It is short for --follow. tail -f on a journal file does not work, because the journal is a binary format — journalctl is the only thing that reads it.

Show more, or less, of what came before

-f starts with the last 10 entries. Change that with -n:

journalctl -f -n 50

Want nothing but new entries? Ask for zero lines of history:

journalctl -f -n 0

That one is worth remembering. Starting a follow from an empty screen makes it obvious which lines your next action produced.

Narrow it before you follow it

A whole machine’s journal scrolls faster than you can read. Cut it down with the same flags you would use for a search — they all work with -f.

They combine:

journalctl -f -u nginx -p warning

Two that people misread. -p warning means warning and worse, not warning exactly — journalctl treats a single priority as a ceiling. And -u takes a unit name, so -u nginx and -u nginx.service are the same thing, but -u /usr/sbin/nginx is not.

Filtering the output live

-g greps inside journalctl itself, and it follows:

journalctl -f -g 'timeout|refused'

It takes a Perl-compatible regular expression. It is case-insensitive as long as your pattern is all lowercase — put one capital in it and the match turns case-sensitive.

-g beats piping to grep because journalctl keeps the entry structure. But if you do pipe, read the next section first.

When the pipe looks like it hung

This prints nothing:

journalctl -f | grep error > /tmp/errors.txt

The file stays empty for minutes. Nothing is broken. grep writes in 4 KB blocks whenever its output is a pipe or a file, so it sits on your matches until it has collected enough of them.

Force it to flush per line:

journalctl -f | grep --line-buffered error > /tmp/errors.txt

The same thing happens one stage later in a longer pipeline. stdbuf -oL applies the fix to a command that has no flag of its own:

journalctl -f | stdbuf -oL cut -d' ' -f5- | grep --line-buffered error

You only see this when the output is redirected. journalctl -f | grep error straight to your terminal works, because grep line-buffers when it is writing to a terminal. That is why the pipeline you tested by hand behaves differently the moment you put it in a script.

When it prints nothing at all

Two causes, and they look identical.

The unit has not logged anything yet. journalctl -f -u myapp on a service that is quiet prints nothing and waits. Confirm the unit name is right:

systemctl list-units --type=service | grep myapp

A misspelled unit name is not an error. journalctl waits for a unit that will never log.

You cannot read the whole journal. Run as an ordinary user, journalctl shows you your own user’s entries and nothing else. Check:

id -nG | tr ' ' '\n' | grep -E 'systemd-journal|adm|wheel'

Add yourself to systemd-journal and log out and back in:

sudo usermod -aG systemd-journal "$USER"

Or use sudo journalctl -f. If neither prints anything, the journal may not be persistent — see clearing journalctl logs and limiting journal disk usage.

Following more than one machine

Here is where -f runs out. It follows the journal on the host you are sitting on. An incident rarely stays on one host.

The usual workaround is a terminal split four ways with four SSH sessions in it. It works until the fifth host, and it does not survive the connection dropping.

Ship the journal to one place instead and follow it there. On each machine, install the CL Agent and point it at your instance:

# /etc/clagent.toml
URL = "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

Then open the source in Central Logging and click Tail. It is the same idea as -f: new rows appear at the top as they land, oldest drop off the bottom.

How live “live” is. Be clear about this, because it is not -f.

• The agent reads the journal and uploads every 30 seconds.
• The tail view asks the server for new rows every 2 seconds.

So a line shows up somewhere between 2 and 32 seconds after the program wrote it. That is fine for watching a deploy across nine hosts. It is not fine for watching a single request go through a single service — for that, SSH in and use journalctl -f.

Two more honest limits. The tail follows one log source at a time, so one source per host means one host at a time; if you want several machines in one stream, point their agents at a shared source and tell them apart with _HOSTNAME. And the tail view has no filter box — it shows everything landing in that source. To narrow it, stop tailing and search.

Searching instead of watching

Watching is the wrong tool for anything older than a few minutes. Once the entries have landed, ask a question instead:

SELECT datetime(timestamp, 'unixepoch') AS at,
       json_extract(msg, '$._HOSTNAME')     AS host,
       json_extract(msg, '$._SYSTEMD_UNIT') AS unit,
       json_extract(msg, '$.MESSAGE')       AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.MESSAGE') LIKE '%refused%'
ORDER BY timestamp DESC
LIMIT 100;

Keep the json_valid(msg) guard in the WHERE clause. json_extract raises an error the first time it meets a line that is not JSON, and one plain-text line in the source is enough to stop the whole query.

timestamp is when the server received the line, not when the event happened. That is close enough while you are following along live. It is not close enough after an outage, when a batch of old entries arrives at once and sorts as if it just happened. Use journald’s own clock for that — it is microseconds, so divide:

SELECT datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000, 'unixepoch') AS at,
       json_extract(msg, '$.MESSAGE') AS message
FROM logs
WHERE json_valid(msg)
ORDER BY at DESC
LIMIT 100;

Keep the CAST. __REALTIME_TIMESTAMP arrives as a JSON string, and SQLite sorts every TEXT value above every INTEGER, so arithmetic and comparisons on it go quietly wrong without one.

Stop watching, get told

The real fix for staring at a follow is not staring at it. Write a rule that fires when the line you are waiting for appears:

SELECT 1 FROM logs
WHERE timestamp > strftime('%s', 'now') - 300
  AND json_valid(msg)
  AND json_extract(msg, '$.MESSAGE') LIKE '%connection refused%'
LIMIT 1;

Alert rules run every 5 minutes and notify when the query returns any row, so SELECT 1 ... LIMIT 1 is all the work it needs to do. Details in alerting and alerting on errors in your logs.

Where to go next

• How to view logs for a systemd service — -u in depth, and what to check when a unit prints nothing.
• How to save journalctl output to a file — the opposite problem: getting the journal out without losing entries.
• Sending journalctl logs to a centralized logging system — the full shipping setup behind the tail view.
• Querying logs with SQL — what to ask once you stop watching.