How to save journalctl output to a file

Introduction

journalctl > logs.txt works. The trouble starts on the second run.

Run it again and you either overwrite what you had or append a second copy of everything. This guide covers the redirect, the flags worth setting before you redirect, and the one flag that makes repeated exports safe.

The short answer

journalctl > /tmp/logs.txt

That is the whole thing. journalctl only pipes through a pager when its output is a terminal. Redirecting to a file means it is not, so there is no less to fight.

Add --no-pager if you want the command to behave identically whether you run it by hand or from a script:

journalctl --no-pager > /tmp/logs.txt

Narrow it before you write it

A full journal is large and most of it is not what you are looking for. Cut it down first.

They combine:

journalctl -u nginx --since "1 hour ago" -p err > /tmp/nginx-errors.txt

Two notes on the ones people misread. -p err means err and worse, not err exactly — journalctl treats a single priority as a ceiling. And -g is case-insensitive as long as your pattern is all lowercase; put a capital in it and the match turns case-sensitive.

Pick a format

The default format is syslog-style local time with no year in it. That is fine for reading and bad for anything else.

Add --utc and every host writes the same clock:

journalctl -u nginx -o short-iso --utc > /tmp/nginx.txt

JSON is bigger than you expect

Every journal entry carries about thirty fields. -o json writes all of them, which makes the file several times the size of -o short-iso. Trim it:

journalctl -o json --output-fields=MESSAGE,_SYSTEMD_UNIT,_HOSTNAME > /tmp/logs.jsonl

__CURSOR, __REALTIME_TIMESTAMP and __MONOTONIC_TIMESTAMP are printed whatever you ask for. journalctl always includes them.

One more thing about JSON. A message containing bytes that are not valid UTF-8 comes out as an array of numbers instead of a string. It is rare. It will still break a parser that assumes MESSAGE is text.

Getting the first lines instead of the last

journalctl prints oldest first, so head gives you the beginning of the journal and -n gives you the end:

journalctl | head -50      # the oldest 50 entries
journalctl -n 50           # the newest 50, oldest first
journalctl -n 50 -r        # the newest 50, newest first

The second run is the problem

> throws away the previous export. >> keeps it and writes a second copy of every entry underneath. Neither is what you want from a job that runs on a schedule.

--cursor-file fixes it. journalctl records where it stopped and starts there next time:

journalctl --cursor-file=/var/lib/journal-export.cursor -o json \
  >> /var/log/journal-export.jsonl

On the first run the cursor file does not exist, so journalctl prints the entire journal and then writes the cursor. Every run after that prints only what arrived since.

To start from now rather than from the beginning of the journal, seed the cursor with a single entry first:

journalctl -r -n 1 -o json --cursor-file=/var/lib/journal-export.cursor > /dev/null

That is exactly what our CL Agent does on startup, for the same reason: nobody wants their first upload to be the whole journal.

The file is not a backup of the journal

Two limits sit on either side of your export.

The journal itself is capped. Check it with journalctl --disk-usage. Trim it with --vacuum-size=500M or --vacuum-time=30d. Once systemd has vacuumed an entry it is gone, and it does not care whether you exported it first.

Your export file is capped by nothing at all. Put it in logrotate before it fills the disk.

Where a file stops working

One host is fine. grep the file, find the line, done.

Ten hosts is ten files, ten clocks and ten grep runs. Tracing one request across a proxy, an app and a database means three files open side by side and timestamps you have to line up by hand. The exports also live on the machine that produced them, which is the machine you lose when the incident is that the machine died.

Sending the same output to a central server instead

The output that goes into a file goes over the network just as easily. Replace the redirect with a pipe:

journalctl -o json --cursor-file=/var/lib/journal-export.cursor \
  | curl -fsS -X POST --data-binary @- \
      "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

-o json puts one JSON object on each line, and the ingest endpoint stores one row per line. The shapes match, so there is nothing to configure.

Compress it if the batches are large or the link is slow:

journalctl -o json --cursor-file=/var/lib/journal-export.cursor | gzip -c \
  | curl -fsS -X POST --data-binary @- \
      "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

Central Logging spots gzip by looking at the first bytes of the body. You do not need a Content-Encoding header, and sending one changes nothing.

Run it on a schedule

Do not put the bare pipe in a crontab. When there is nothing new, journalctl prints nothing and curl posts an empty body every minute. Wrap it:

#!/bin/sh
# /usr/local/bin/journal-ship
set -eu

CURSOR=/var/lib/journal-export.cursor
INGEST="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

OUT=$(journalctl -o json --cursor-file="$CURSOR")
[ -n "$OUT" ] || exit 0

printf '%s\n' "$OUT" | gzip -c \
  | curl -fsS -X POST --data-binary @- "$INGEST"

* * * * * /usr/local/bin/journal-ship

The honest caveat. --cursor-file moves the cursor when journalctl runs, not when curl succeeds. If the POST fails, those entries are not sent and not retried — the cursor has already passed them. The agent has the same property. For most people a one-minute gap during an outage is acceptable; if it is not for you, buffer the output to a file first and delete it only after a 2xx.

Once the logs are landing, the CL Agent does all of the above for you every 30 seconds, and ships host and process metrics on the same connection.

Querying what you sent

The timestamp column is when the server received the line, not when the event happened. After an outage a batch of old entries all arrive at once and sort as if they just occurred. Use journald’s own clock instead — it is microseconds, so divide:

SELECT datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000, 'unixepoch') AS at,
       json_extract(msg, '$._HOSTNAME')     AS host,
       json_extract(msg, '$._SYSTEMD_UNIT') AS unit,
       json_extract(msg, '$.MESSAGE')       AS message
FROM logs
WHERE json_valid(msg)
ORDER BY at DESC
LIMIT 50;

Keep the json_valid(msg) guard. json_extract raises an error the first time it meets a line that is not JSON, and one plain-text line in the source is enough to stop the query.

Where to go next

• Viewing logs for a specific systemd service — -u, -f, and what to check when the command prints nothing.
• Sending journalctl logs to a centralized logging system — the full shipping setup, including the agent.
• Querying logs — the SQL side, once the logs have landed.