How to search journalctl logs for a specific string

Introduction

journalctl -g "connection refused"

-g searches the journal for a pattern. It is faster than piping to grep and it keeps the journal’s own formatting, but it has a case rule that surprises people and a blind spot that sends them looking in the wrong place.

The short answer

journalctl -g "connection refused"          # everywhere
journalctl -u nginx -g "upstream"           # one unit
journalctl -g "timeout" --since "-1h"       # the last hour

-g is short for --grep. The pattern is a Perl-compatible regular expression, not a plain substring, so the full syntax is available:

journalctl -g "error|fatal|panic"
journalctl -g "^Failed password"
journalctl -g "port [0-9]+"

The case rule

This one is worth committing to memory, because nothing warns you.

If your pattern is entirely lowercase, matching is case insensitive. Put a single uppercase character in it and matching becomes case sensitive.

journalctl -g "error"      # matches error, Error, ERROR
journalctl -g "Error"      # matches Error only

So a search that was finding everything stops finding most of it the moment you capitalise a word. It looks like the log stopped containing the string.

Override it explicitly when you care:

journalctl -g "Error" --case-sensitive=false    # matches error, Error, ERROR
journalctl -g "error" --case-sensitive=true     # matches error only

In a script, always pass --case-sensitive. Relying on the shape of the pattern is how a monitoring job quietly stops matching.

What -g cannot match

-g filters on the MESSAGE field. Only that field. It does not see the unit name, the hostname, the PID or any other metadata, even though journalctl prints them right there on the same line.

So this does not work the way it reads:

journalctl -g "nginx"      # finds "nginx" in message text only

It will not find every message from nginx.service. It finds lines that happen to say “nginx” in the text.

To match on metadata, use a field match instead:

journalctl _SYSTEMD_UNIT=nginx.service
journalctl _HOSTNAME=web-01
journalctl _PID=4412
journalctl SYSLOG_IDENTIFIER=sshd

Field matches are exact, not substrings — _SYSTEMD_UNIT=nginx matches nothing, because the value is nginx.service. To see the values a field actually takes:

journalctl -F _SYSTEMD_UNIT

And the two combine, which is usually what you actually wanted:

journalctl _SYSTEMD_UNIT=nginx.service -g "upstream"

The ordering surprise

journalctl normally prints oldest first. Combining -g with -n changes that silently.

When --grep is used with --lines and the count is not prefixed with +, --reverse is implied. journalctl searches backwards from the end of the journal, because it cannot know how many matches exist until it has looked. So the output arrives newest first:

journalctl -g "timeout" -n 20      # newest 20 matches, newest first
journalctl -g "timeout" -n +20     # oldest 20 matches, oldest first

--follow is the exception: -g "timeout" -f does not flip the order.

If a script parses this output and assumes chronological order, -n is the flag that breaks it.

When -g is not available

-g needs PCRE2, which is an optional dependency loaded at runtime. On a minimal build or a container image without the library, you get:

PCRE2 support is not compiled in.

Fall back to grep. Two flags make it behave:

journalctl -o cat | grep "connection refused"
journalctl -f | grep --line-buffered "connection refused"

-o cat prints the message text with no timestamp or unit prefix, so grep matches your text and not the metadata around it — the same scope -g has. Drop it if you want the metadata back.

--line-buffered matters only with -f. When grep’s output is a pipe rather than a terminal it buffers in blocks, so matches appear in bursts minutes late and the command looks hung. This is the single most common reason a journalctl -f | grep pipeline appears to do nothing.

grep -C 3 is the other reason to reach for it. -g has no context flag, so when you want the three lines either side of a match, pipe it:

journalctl -o short-iso | grep -C 3 "Out of memory"

Searching more than one machine

Every command above searches one journal on one host.

The searches that matter usually do not respect that boundary. A request ID appears in the proxy, the application and the database, on three machines. An attacker’s IP appears in the auth log of whichever hosts they reached, and you do not know which those are until you have searched all of them. for h in ...; do ssh $h journalctl -g ...; done works until a host is down, the output interleaves badly, or you want to search last month on a machine that has since been rebuilt.

Shipping journals to one place makes it a single query. The CL Agent forwards journald with the fields intact, so _HOSTNAME and _SYSTEMD_UNIT survive and you can still tell the hosts apart afterwards.

Searching once the logs have landed

Central Logging stores each line in SQLite with a full-text index over it. The search box uses that index, and so can a query:

SELECT datetime(logs.timestamp, 'unixepoch') AS received,
       logs.source,
       logs.msg
FROM logs_fts
JOIN logs ON logs.id = logs_fts.rowid
WHERE logs_fts MATCH 'timeout OR refused'
ORDER BY rank
LIMIT 50;

logs_fts is an external-content table, so it stores no second copy of your logs. Join back to logs on rowid = id to get the columns. ORDER BY rank sorts by relevance; ORDER BY logs.timestamp DESC sorts by time.

Four things about that index are worth knowing before you trust a result.

Use the table name, not an alias. FROM logs_fts f ... WHERE f MATCH ... fails with no such column: f. The MATCH operator wants the real table name even when the table is aliased.

There is no stemming. MATCH 'timeout' does not match the line “upstream timed out”, and MATCH 'timed' does not match “timeout”. FTS5 matches tokens as written. Search for both when the distinction matters:

WHERE logs_fts MATCH 'timeout OR timed'

The index covers the whole stored line, including JSON field names. A journald entry arrives as a JSON object, so MATCH 'MESSAGE' matches every journald row in the source — it is matching the key, not anything anyone logged. Usually this helps: MATCH 'nginx' finds lines whose _SYSTEMD_UNIT is nginx.service even when the text never says nginx, which is exactly the thing -g could not do. Occasionally it produces a match you did not intend.

Quote a phrase to keep it together. The tokenizer splits on punctuation, so an IP address becomes several tokens:

WHERE logs_fts MATCH '"203.0.113.9"'

Without the quotes you get every line containing those numbers separately.

Matching a field exactly instead

When you want a field rather than free text, skip the index and read the JSON:

SELECT datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000, 'unixepoch') AS at,
       json_extract(msg, '$._HOSTNAME') AS host,
       json_extract(msg, '$.MESSAGE')   AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$._SYSTEMD_UNIT') = 'nginx.service'
ORDER BY at DESC
LIMIT 50;

Keep the json_valid(msg) guard. json_extract raises an error on the first line that is not JSON, and one plain-text line in the source is enough to stop the query. The guard protects the terms after it in WHERE, because SQLite short-circuits AND — it does not protect the SELECT list, so it has to eliminate the bad rows rather than merely precede them.

Turning a search into an alert

A search you run twice is a search worth automating. Central Logging evaluates a rule by running the query and checking whether anything came back, so the alert is your search with the columns dropped:

SELECT 1 FROM logs_fts WHERE msg MATCH 'panic OR fatal' LIMIT 1;

Set it to alert on results. Rules run every five minutes.

Add a time bound if you want it to fire on new matches rather than on the whole retained history:

SELECT 1 FROM logs
WHERE timestamp > CAST(strftime('%s', 'now', '-10 minutes') AS INTEGER)
  AND msg LIKE '%panic%'
LIMIT 1;

That second one uses LIKE rather than the index, because combining a MATCH with a time window means joining back to logs anyway. LIKE is slower and fine at this scale. Cast the strftime result — it returns text, and comparing text against a number is a silent way to match nothing.

Where to go next

• How to show only errors in journalctl — -p err, and why searching the text is often the only thing that works.
• How to view journalctl logs for a specific time range — --since and --until to bound the search.
• How to view logs for a systemd service — -u in depth, and what to check when a unit prints nothing.
• How to follow logs in real time with journalctl -f — searching a stream as it arrives.
• How to query logs with SQL — the full SQL side, including the traps that return wrong answers quietly.
• Querying logs — the reference.