journalctl cursor: read only new entries with --cursor-file and --after-cursor

A journalctl cursor is a bookmark. It is also the place most home-made log shippers lose entries.

Run journalctl with a cursor and it prints only what arrived after the bookmark. That is how a cron job avoids sending the same line twice. The bookmark can also move before your upload has worked, point at nothing, or never be saved at all. This guide covers the flags, the six traps, and a script that avoids all of them.

The short answer

journalctl -q --cursor-file=/var/lib/myjob.cursor

The first run prints the journal and saves the position of the last entry in the file. Every run after that prints only the entries that arrived since.

Keep -q. Without it, a run with nothing new prints -- No entries -- to stdout, and a script cannot tell that from a log line.

What a cursor is

A cursor is a string that names one journal entry:

s=0639e4…;i=5a3f1;b=8c1d…;m=2f4b1c7;t=63f1a2b3c4d5e;x=9e2a…

It holds the journal file’s sequence number, the boot ID, both timestamps and a hash of the entry. The systemd man page calls the format “private and subject to change”. Store it and hand it back. Do not parse it.

You can get one two ways. --show-cursor prints it after the last entry:

journalctl -n 1 --show-cursor

Oct 13 09:14:02 web1 sshd[812]: Accepted publickey for deploy from 10.0.0.5
-- cursor: s=0639e4…;i=5a3f1;b=8c1d…;m=2f4b1c7;t=63f1a2b3c4d5e;x=9e2a…

And every entry in -o json carries its own cursor in the __CURSOR field. A script reading JSON does not need --show-cursor.

The four flags

--cursor-file arrived in systemd 242. RHEL 8 and its rebuilds ship 239, so they do not have it. Use --after-cursor there and save the cursor yourself, as in the script below.

Six things that repeat or lose entries

1. No cursor means the whole journal

--cursor-file with no file falls back to your other options. With no other options, that is the entire journal, from the oldest entry. An empty file counts as no file. So touch-ing the file first does not help.

Start from now instead. Seed the file once, before the first real run:

journalctl -n 1 --cursor-file=/var/lib/myjob.cursor > /dev/null

That prints the newest entry into /dev/null and saves its cursor. The next run starts after it.

2. “– No entries –” goes to stdout

When nothing new has arrived, journalctl prints -- No entries --. It goes to stdout, not stderr, and it ignores -o json.

So journalctl -o json --cursor-file=F | curl … posts that line once a minute on an idle host, and your log server stores it as a log line. -q turns it off. Check whether you already have it:

SELECT COUNT(*) FROM logs
WHERE msg = '-- No entries --'
  AND timestamp > CAST(strftime('%s', 'now', '-1 day') AS INTEGER);

3. The cursor moves when journalctl finishes, not when the upload works

--cursor-file saves the position the moment journalctl exits. In a pipe to curl, that is before curl has heard back from the server. A failed upload is never retried, because the cursor has already moved past it.

That is fine for a homelab where a one-minute gap costs nothing. Anywhere else, drop --cursor-file and use --after-cursor. Save the new cursor yourself, only after the server says yes. The script at the end of this guide does exactly that.

4. With -f, the cursor is saved only when journalctl exits cleanly

journalctl -f --cursor-file=F saves nothing while it runs. It writes the file once, on the way out.

From systemd 254, journalctl catches SIGINT and SIGTERM and saves the cursor before it exits. So Ctrl-C and systemctl stop both save it. Before 254, nothing caught those signals. The file is never written in follow mode, and every restart begins from wherever the file last pointed.

SIGKILL, the OOM killer or a crash skip the save on every version. The next start replays everything since the last clean exit. You get duplicates, not a hole, but on a busy host that is a lot of duplicates.

A shipper that runs every minute and exits does not have this problem. That is one more reason to prefer a timer over a long-running -f.

5. –since and a cursor do not mix

--since with --cursor or --after-cursor has always been an error. From systemd 256, --cursor-file joins them:

Please specify only one of --since=, --cursor=, --cursor-file=, and --after-cursor=.

Before 256, --since with --cursor-file is accepted, and it does something odd. With no cursor in the file, --since applies. Once the file has a cursor, --since is ignored without a word. A script written on Debian 12 that seeds with --since "-1h" stops working on Debian 13.

Seed with -n 1 instead. It works on every version.

6. A cursor can point at an entry that no longer exists

journald vacuums old journal files to stay inside its size limit. If the entry your cursor names was in one of them, journalctl does not complain. It finds the next entry that still exists and carries on. Whatever was vacuumed in between is gone, and nothing tells you. Keep the journal large enough to cover your longest outage. See limiting journal disk usage.

A cursor that will not parse is louder. journalctl prints Failed to seek to cursor: Invalid argument and exits non-zero, on every run, until you delete the file.

One more, on systemd 255 and older. --after-cursor skips one entry after seeking, whether or not it was the cursor’s own entry. If the cursor came from an unfiltered run and you now add -u nginx, the cursor’s entry is not an nginx entry. journalctl lands on the first nginx entry after it and skips that one instead. Systemd 256 checks before it skips. On older versions, use the same filters to seed the cursor as you use to read after it.

-n with a cursor means the next N, not the last N

Without a cursor, -n 100 shows the last 100 entries. With one, it shows the first 100 after the cursor, and --cursor-file saves the 100th.

That makes -n a batch size. A shipper that was down for a day can catch up 5,000 entries at a time instead of building one enormous request:

journalctl -q -o json -a -n 5000 --after-cursor "$(cat /var/lib/myjob.cursor)"

A shipper that never loses a batch

This is the version to put in cron. It holds the cursor itself and moves it only after the server answers 2xx.

#!/bin/sh
# /usr/local/bin/journal-ship
set -eu

CURSOR=/var/lib/journal-export.cursor
INGEST="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

# resume after the last entry the server accepted, 5000 entries at a time;
# with no cursor yet, start from the newest entry instead of the whole journal
if [ -s "$CURSOR" ]; then
  OUT=$(journalctl -q -o json -a -n 5000 --after-cursor "$(cat "$CURSOR")")
else
  OUT=$(journalctl -q -o json -a -n 1)
fi
[ -n "$OUT" ] || exit 0

printf '%s\n' "$OUT" | gzip -c \
  | curl -fsS -X POST --data-binary @- "$INGEST" > /dev/null

# only now is it safe to move the cursor
NEXT=$(printf '%s\n' "$OUT" | tail -n 1 \
  | sed -n 's/.*"__CURSOR"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p')
[ -n "$NEXT" ] || exit 1
printf '%s\n' "$NEXT" > "$CURSOR.tmp"
mv "$CURSOR.tmp" "$CURSOR"

* * * * * /usr/local/bin/journal-ship

What each part is for:

• -q stops -- No entries -- reaching the server, and keeps it out of $OUT, so an idle minute posts nothing.
• -n 5000 caps a batch after an outage. The next run takes the next 5,000.
• -a stops journalctl replacing any field over 4096 bytes with null. See JSON output from journalctl.
• curl -f makes an HTTP 500 a failure. With set -e the script stops there, and the cursor stays where it was.
• [ -n "$NEXT" ] || exit 1 refuses to save an empty cursor. An empty file means “start from the top of the journal” next time.
• The .tmp and mv mean a crash mid-write cannot leave half a cursor behind.

Keep the cursor in /var/lib, not /tmp. Many distributions clear /tmp on boot, and the first run after a reboot is the one that should catch the boot’s own lines.

We ran this script against a stand-in journalctl and an ingest server told to answer 500. Idle runs posted nothing. Failed uploads left the cursor alone and went out again on recovery. A 15-entry backlog drained in batches with nothing missing.

The trade is duplicates for holes. If the server stores a batch and the reply is lost, the next run sends that batch again.

Find the entries that arrived twice

Every entry carries its own __CURSOR, so duplicates are easy to count once the logs have landed in Central Logging:

SELECT json_extract(msg, '$.__CURSOR') AS cursor, COUNT(*) AS copies
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.__CURSOR') IS NOT NULL
  AND timestamp > CAST(strftime('%s', 'now', '-1 day') AS INTEGER)
GROUP BY cursor
HAVING COUNT(*) > 1
ORDER BY copies DESC
LIMIT 20;

Keep json_valid(msg) first. One plain-text row in the source is enough to stop json_extract with malformed JSON.

A few duplicates after an outage are the retry doing its job. Thousands at a regular interval point to trap 4: a -f shipper restarting without saving its cursor.

Or let the agent hold the cursor

The CL Agent does all of this for you every 30 seconds. It never uses --cursor-file. It reads its saved cursor, passes it as --after-cursor, caps each batch, and saves the new cursor only after the server accepts the upload. The cursor lives in /var/lib/clagent/clagent-cursor.txt.

One honest difference from the script above. If journalctl rejects the saved cursor, the agent does not stop. It starts again from the newest entry and logs the saved cursor was unusable. You keep shipping, but anything between the old cursor and now is skipped.

Where to go next

• How to save journalctl output to a file — the same cursor pointed at a file instead of the network.
• Sending journalctl logs to a centralized logging system — the shipping side in full, including the Go loop the agent uses.
• How to get JSON output from journalctl — the format this script sends, and why -a matters.
• How to show the last 100 lines in journalctl — -n without a cursor.
• How to view journalctl logs for a specific time range — --since and --until, when a time is what you have.
• How to clear journalctl logs and limit journal disk usage — the vacuum that can delete the entry your cursor points at.
• How to find out why a Linux server rebooted — why the first run after a reboot matters.
• How to read journal files from another server — what is left when nothing was shipped.