Sending journalctl logs to a centralized logging system

Introduction

The logs you want after an incident are on the machine that had the incident.

journalctl reads them beautifully, right up until the host is gone. This guide covers moving journald entries to a central server as they happen, and the one ordering mistake that makes the whole pipeline lose exactly the minute you care about.

Get the entries as JSON

Central Logging stores one row per line and parses JSON, so -o json is the format to send.

journalctl -o json -a

One object per line. -a matters: without it journalctl replaces any field over 4096 bytes with null, and a stack trace clears 4 KB without trying. Getting JSON output from journalctl covers that and the two other cases where a value is not a string.

Reading one unit rather than the whole journal? See viewing logs for a specific systemd service.

Sending only new logs

From the journalctl -h:

--cursor-file=FILE Show entries after cursor in FILE and update FILE

--cursor-file=/tmp/my-cursor.txt keeps a record of what has already been read. journalctl stores the ID of the last entry it printed in that file. The next run reads the ID back and starts after it.

From here you have all the parts you need. The simplest version is a one minute cron job that pipes the output through curl to your Central Logging instance over HTTPS.

One minute cron:

* * * * * journalctl -o json -a --cursor-file=/path/to/your/cursor-file | curl -fsS -X POST --data-binary @- https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN

Read this before you rely on it. --cursor-file updates the cursor when journalctl reads the entries, not when curl delivers them. Put those two in a pipeline and the cursor moves whether or not the upload worked. One restart of your instance, one dropped connection, and that minute of logs is gone from the server’s copy for good.

Keep the pipeline for a homelab where a gap costs nothing. For anything you would go looking for after an incident, hold the cursor yourself and move it only after a 2xx — the journal export guide has the shell version, and the Go below does the same thing.

Use --data-binary, not -d. curl strips newlines out of -d @-, and the ingest endpoint stores one row per line. With -d a batch of seven journal entries arrives as a single row with all seven jammed together. With --data-binary it arrives as seven.

This is roughly what the CL Agent does, in Go. Note what it does not do: it never passes --cursor-file. It reads the saved cursor, hands it to journalctl as --after-cursor, and writes the new one only after the server answers 2xx.

// shipJournal sends everything the server has not acknowledged yet. It stops
// at the first batch that fails and leaves the cursor where it was, so the
// next tick starts from the same entry.
func shipJournal(client *http.Client) {
	for {
		cursor := readCursor()

		// journalctl -o json --after-cursor <cursor>
		entries, err := readJournalAfter(cursor, maxBytesPerPost)
		if err != nil || len(entries) == 0 {
			return
		}

		// every entry carries its own cursor in __CURSOR, so the last
		// entry in the batch is the position to save
		next := lastCursor(entries)
		if next == "" || next == cursor {
			return
		}

		// anything other than a 2xx is a failure: leave the cursor alone
		// and these same entries go out again next time
		if err := postLogs(client, conf.URL, entries); err != nil {
			log.Println("sending logs:", err)
			return
		}

		writeCursor(next)
	}
}

The loop matters as much as the ordering. Because the cursor stays put while the server is unreachable, the backlog after an outage can be larger than one request should carry, so readJournalAfter caps each batch and the loop sends as many batches as it takes to catch up.

Two things you get in exchange. An upload whose reply is lost is sent again, so an entry can land twice — a duplicate rather than a hole. And a bad source token now stalls the agent instead of quietly discarding logs, which is the behaviour you want: check journalctl -u clagent and you will see invalid user token every 30 seconds.

Where to go next

• How to save journalctl output to a file — the same flags pointed at a file instead of the network, and what --cursor-file does on its first run.
• How to get JSON output from journalctl — the format this guide sends, and the three cases where a field is not a string.
• How to read journal files from another server — what you are left doing when nothing was shipped.
• How to view logs for a specific systemd service — reading one unit locally, and the SQL that does the same thing across every host once the logs have landed.
• How to clear journalctl logs and limit journal disk usage — capping the local journal once the entries are safely somewhere else.
• How to find out why a Linux server rebooted — where the cursor file lives decides whether you get the boot lines.
• CL Agent — skip the cron job and let the agent do this.