How to clear journalctl logs and limit journal disk usage

Introduction

/var/log/journal is 3.8GB and you never put it there.

journald did. It is allowed to take a share of the disk, and on a busy host it takes all of it. This guide covers checking how much it holds, clearing it now, capping it so it stays capped, and the part people skip — keeping the entries before you delete them.

How much is it actually using

journalctl --disk-usage

Archived and active journals take up 3.8G in the file system.

That is the number journald manages. du -sh /var/log/journal gives the same answer from the other side, and disagreeing by a few megabytes is normal.

If /var/log/journal does not exist, your journal is in /run/log/journal, which is a tmpfs. It is using RAM, not disk, and a reboot empties it.

Clear it now

Two commands, in this order:

sudo journalctl --rotate
sudo journalctl --vacuum-time=1s

The order matters. --vacuum-* only deletes archived journal files. The file journald is currently writing is never a candidate. --rotate closes that file and archives it, which is what makes the next command able to remove it.

Run the vacuum on its own and you will watch --disk-usage barely move, because everything recent is still in the active file.

Pick whichever limit fits:

They combine, and journald applies all of them.

You need root for any of this. As a normal user journalctl operates on your own user journal, so the command succeeds and frees almost nothing.

Stop it filling up again

Vacuuming is a one-off. The cap belongs in /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
SystemKeepFree=2G
SystemMaxFiles=10

sudo systemctl restart systemd-journald
journalctl --disk-usage

What each one does:

• SystemMaxUse — the ceiling for all journal files together. Default is 10% of the filesystem the journal sits on.
• SystemKeepFree — space journald refuses to eat into, whatever SystemMaxUse says. Default is 15% of the filesystem. This is the setting that saves you when the journal shares a partition with your database.
• SystemMaxFiles — how many archived files to keep. Vacuuming works a whole file at a time, so this is the real granularity of “how much do I lose when it trims”.
• SystemMaxFileSize — the size of one file. Smaller files mean finer trimming and more of them.

The System* settings apply to /var/log/journal. The matching Runtime* settings apply to /run/log/journal. Set both if you are not sure which one your host uses.

Do not trust the percentage arithmetic. Set the value, restart, and read journalctl --disk-usage back.

Persistent or not

[Journal]
Storage=persistent

Storage= decides where the journal lives.

auto is the one that surprises people. The directory decides, not the config. Debian has shipped for years without /var/log/journal, so a stock Debian box keeps no journal across reboots and nobody notices until the day they need last week’s logs.

Check yours:

ls -d /var/log/journal 2>/dev/null || echo "volatile — logs die at reboot"

The messages journald throws away on its own

Disk is not the only limit. journald rate-limits each service:

[Journal]
RateLimitIntervalSec=30s
RateLimitBurst=10000

Past the burst inside the interval, journald drops the rest and writes a line saying how many it suppressed. Search for it:

journalctl -g "Suppressed" --since "1 day ago"

A service in a crash loop hits this instantly, so the exact window you want is the window that got dropped. Raise the burst, or set both to 0 to disable rate limiting for that host, and remember you have just removed the thing keeping one chatty service from filling the disk.

Vacuuming is deletion

There is no undo, no archive, no “are you sure”. --vacuum-time=30d on a host you are debugging removes the 31-day-old entries that explain the problem.

The uncomfortable version: you are capping the journal because the host produces a lot of logs, and the hosts that produce a lot of logs are the ones worth reading. The cap and the value are in direct conflict as long as both live on the same disk.

They stop conflicting when the logs are somewhere else.

Ship first, then vacuum freely

Send the journal to a central server and the local cap is a housekeeping setting instead of a retention policy. Trim to 200MB. The entries still exist.

The one-line version:

journalctl -o json --cursor-file=/var/lib/journal-ship.cursor \
  | curl -fsS -X POST --data-binary @- \
      "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

--cursor-file records where journalctl stopped, so the next run sends only what arrived since. -o json puts one JSON object per line, and the ingest endpoint stores one row per line, so the shapes already match.

Use --data-binary, not -d. curl strips newlines out of -d @-, and a batch of forty entries arrives as one row with all forty jammed together.

Order the two operations correctly. Ship, then vacuum. A cron job that vacuums at 03:00 and ships at 03:05 loses whatever fell inside the window. If you run both on a schedule, put them in one script:

#!/bin/sh
# /usr/local/bin/journal-ship-and-trim
set -eu

CURSOR=/var/lib/journal-ship.cursor
INGEST="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

OUT=$(journalctl -o json --cursor-file="$CURSOR")
if [ -n "$OUT" ]; then
  printf '%s\n' "$OUT" | gzip -c | curl -fsS -X POST --data-binary @- "$INGEST"
fi

journalctl --rotate
journalctl --vacuum-size=200M

Central Logging detects gzip from the first bytes of the body. No Content-Encoding header is needed, and sending one changes nothing.

The honest caveat. --cursor-file moves the cursor when journalctl runs, not when curl succeeds. If the POST fails, the script vacuums entries that were never delivered. The set -eu above does not protect you, because curl is inside an if. If that matters for your host, drop the vacuum from this script and run it separately once you trust the upload.

The CL Agent does the shipping half every 30 seconds and keeps its cursor in /tmp/clagent-cursor.txt. It does not vacuum anything — that stays your decision.

Find out what is filling the journal

Once the logs have landed, the question “which host is producing all this” becomes a query rather than a tour of ten machines:

SELECT json_extract(msg, '$._HOSTNAME') AS host,
       COUNT(*)         AS lines,
       SUM(LENGTH(msg)) AS bytes
FROM logs
WHERE json_valid(msg)
  AND timestamp > CAST(strftime('%s', 'now', '-1 day') AS INTEGER)
GROUP BY host
ORDER BY bytes DESC;

Then narrow to the unit responsible:

SELECT json_extract(msg, '$._SYSTEMD_UNIT') AS unit,
       COUNT(*) AS lines
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$._HOSTNAME') = 'web1'
  AND timestamp > CAST(strftime('%s', 'now', '-1 day') AS INTEGER)
GROUP BY unit
ORDER BY lines DESC
LIMIT 20;

Keep the json_valid(msg) guard in both. json_extract raises an error the first time it meets a line that is not JSON, and one plain-text line in the source is enough to stop the query.

The timestamp column is when the server received the line, not when the event happened. For a volume question that is the right clock — you want to know what arrived. For event order, use journald’s own timestamp instead. See querying logs with SQL.

Where to go next

• How to save journalctl output to a file — the flags worth setting before you export, and what --cursor-file does on its first run.
• How to find out why a Linux server rebooted — what a volatile journal costs you the morning after.
• Sending journalctl logs to a centralized logging system — the full shipping setup, including the agent.
• Querying logs — the reference for the SQL side.