How to save journalctl output to a file

Introduction

journalctl > logs.txt works. The trouble starts on the second run.

Run it again and you either overwrite what you had or append a second copy of everything. This guide covers the redirect, the flags worth setting before you redirect, and the one flag that makes repeated exports safe.

The short answer

journalctl > /tmp/logs.txt

That is the whole thing. journalctl only pipes through a pager when its output is a terminal. Redirecting to a file means it is not, so there is no less to fight.

Add --no-pager if you want the command to behave identically whether you run it by hand or from a script:

journalctl --no-pager > /tmp/logs.txt

Narrow it before you write it

A full journal is large and most of it is not what you are looking for. Cut it down first.

They combine:

journalctl -u nginx --since "1 hour ago" -p err > /tmp/nginx-errors.txt

Two notes on the ones people misread. -p err means err and worse, not err exactly — journalctl treats a single priority as a ceiling. And -g is case-insensitive as long as your pattern is all lowercase; put a capital in it and the match turns case-sensitive.

Pick a format

The default format is syslog-style local time with no year in it. That is fine for reading and bad for anything else.

Add --utc and every host writes the same clock:

journalctl -u nginx -o short-iso --utc > /tmp/nginx.txt

JSON is bigger than you expect

Every journal entry carries about thirty fields. -o json writes all of them, which makes the file several times the size of -o short-iso. Trim it:

journalctl -o json -a --output-fields=MESSAGE,_SYSTEMD_UNIT,_HOSTNAME > /tmp/logs.jsonl

__CURSOR, __REALTIME_TIMESTAMP and __MONOTONIC_TIMESTAMP are printed whatever you ask for. journalctl always includes them. So is _BOOT_ID.

Keep the -a. Without it journalctl writes null in place of any field where the name, an = and the value together reach 4096 bytes — a stack trace goes in and an empty message comes out, with no warning and no non-zero exit. See getting JSON output from journalctl for that and the two other cases where a value is not a string.

One more thing about JSON. A message containing bytes that are not valid UTF-8 comes out as an array of numbers instead of a string. It is rare. It will still break a parser that assumes MESSAGE is text.

Getting the first lines instead of the last

journalctl prints oldest first, so head gives you the beginning of the journal and -n gives you the end:

journalctl | head -50      # the oldest 50 entries
journalctl -n 50           # the newest 50, oldest first
journalctl -n 50 -r        # the newest 50, newest first

The second run is the problem

> throws away the previous export. >> keeps it and writes a second copy of every entry underneath. Neither is what you want from a job that runs on a schedule.

--cursor-file fixes it. journalctl records where it stopped and starts there next time:

journalctl --cursor-file=/var/lib/journal-export.cursor -o json -a \
  >> /var/log/journal-export.jsonl

On the first run the cursor file does not exist, so journalctl prints the entire journal and then writes the cursor. Every run after that prints only what arrived since.

To start from now rather than from the beginning of the journal, seed the cursor with a single entry first:

journalctl -r -n 1 -o json --cursor-file=/var/lib/journal-export.cursor > /dev/null

That is exactly what our CL Agent does on startup, for the same reason: nobody wants their first upload to be the whole journal.

The file is not a backup of the journal

Two limits sit on either side of your export.

The journal itself is capped. Check it with journalctl --disk-usage. Trim it with --vacuum-size=500M or --vacuum-time=30d. Once systemd has vacuumed an entry it is gone, and it does not care whether you exported it first. See clearing journalctl logs and limiting journal disk usage for the caps and the order to run things in.

Your export file is capped by nothing at all. Put it in logrotate before it fills the disk.

Where a file stops working

One host is fine. grep the file, find the line, done.

Ten hosts is ten files, ten clocks and ten grep runs. Tracing one request across a proxy, an app and a database means three files open side by side and timestamps you have to line up by hand. The exports also live on the machine that produced them, which is the machine you lose when the incident is that the machine died.

Sending the same output to a central server instead

The output that goes into a file goes over the network just as easily. Replace the redirect with a pipe:

journalctl -o json -a --cursor-file=/var/lib/journal-export.cursor \
  | curl -fsS -X POST --data-binary @- \
      "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

-o json puts one JSON object on each line, and the ingest endpoint stores one row per line. The shapes match, so there is nothing to configure.

Compress it if the batches are large or the link is slow:

journalctl -o json -a --cursor-file=/var/lib/journal-export.cursor | gzip -c \
  | curl -fsS -X POST --data-binary @- \
      "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

Central Logging spots gzip by looking at the first bytes of the body. You do not need a Content-Encoding header, and sending one changes nothing.

Run it on a schedule

Do not put the bare pipe in a crontab. When there is nothing new, journalctl prints nothing and curl posts an empty body every minute. Wrap it:

#!/bin/sh
# /usr/local/bin/journal-ship
set -eu

CURSOR=/var/lib/journal-export.cursor
INGEST="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

OUT=$(journalctl -o json -a --cursor-file="$CURSOR")
[ -n "$OUT" ] || exit 0

printf '%s\n' "$OUT" | gzip -c \
  | curl -fsS -X POST --data-binary @- "$INGEST"

* * * * * /usr/local/bin/journal-ship

The honest caveat. --cursor-file moves the cursor when journalctl runs, not when curl succeeds. If the POST fails, those entries are not sent and not retried — the cursor has already passed them. For most people a one-minute gap during an outage is acceptable. If it is not, keep the cursor yourself and advance it only after a 2xx. That is the next section.

Retry without losing a batch

Drop --cursor-file. Read the cursor, pass it to journalctl as --after-cursor, and write the new one only when curl comes back clean.

#!/bin/sh
# /usr/local/bin/journal-ship
set -eu

CURSOR=/var/lib/journal-export.cursor
INGEST="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

# resume where the last successful upload finished
if [ -s "$CURSOR" ]; then
  set -- --after-cursor "$(cat "$CURSOR")"
else
  set --
fi

OUT=$(journalctl -o json -a "$@")
[ -n "$OUT" ] || exit 0

printf '%s\n' "$OUT" | gzip -c \
  | curl -fsS -X POST --data-binary @- "$INGEST" > /dev/null

# only now is it safe to move the cursor
printf '%s\n' "$OUT" | tail -n 1 \
  | sed -n 's/.*"__CURSOR"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' > "$CURSOR"

Every entry journalctl -o json prints carries its own cursor in the __CURSOR field, so you do not need --show-cursor to find it. The last entry’s cursor is the one to save.

curl -f is what makes this work. Without -f, curl exits 0 on an HTTP 500 and the script would record the cursor for a batch the server rejected. set -eu then stops the script before the cursor line runs.

Two things to accept in exchange. A batch whose reply is lost gets sent again, so an entry can land twice — a duplicate rather than a hole. And a server that stays down builds a backlog, so the upload after a long outage is large; cap it if that matters to you.

Once the logs are landing, the CL Agent does all of the above for you every 30 seconds, and ships host and process metrics on the same connection. It does keep its own cursor and only advances it once the server has taken the entries, so a failed send costs it a retry rather than the batch.

Querying what you sent

The timestamp column is when the server received the line, not when the event happened. After an outage a batch of old entries all arrive at once and sort as if they just occurred. Use journald’s own clock instead — it is microseconds, so divide:

SELECT datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000, 'unixepoch') AS at,
       json_extract(msg, '$._HOSTNAME')     AS host,
       json_extract(msg, '$._SYSTEMD_UNIT') AS unit,
       json_extract(msg, '$.MESSAGE')       AS message
FROM logs
WHERE json_valid(msg)
ORDER BY at DESC
LIMIT 50;

Keep the json_valid(msg) guard. json_extract raises an error the first time it meets a line that is not JSON, and one plain-text line in the source is enough to stop the query.

Where to go next

• How to get JSON output from journalctl — -o json, and the three cases where a field is not a string.
• How to read journal files from another server — --file, -D and --root for a journal you copied off another box.
• Viewing logs for a specific systemd service — -u, -f, and what to check when the command prints nothing.
• Viewing journalctl logs for a specific time range — --since and --until in full, and the midnight trap.
• Showing only errors in journalctl — -p err, and why it often finds nothing.
• Searching journalctl logs for a specific string — -g, and the fields it cannot match.
• Sending journalctl logs to a centralized logging system — the full shipping setup, including the agent.
• Querying logs with SQL — what to ask the logs once they have landed.
• Querying logs — the reference for the SQL side.