du says the directory is empty. df says the disk is full. Both are telling the truth.
That is the case most guides skip, and it is the one that wastes an afternoon. Start with the easy version anyway, because most of the time the easy version is the answer.
sudo du -xh /var/log | sort -rh | head -20
-x keeps du on one filesystem, so a bind mount or a network share under /var/log does not send it wandering. sort -rh sorts the human-readable sizes properly.
Then the individual files, not just the directories:
sudo find /var/log -type f -printf '%s\t%p\n' | sort -rn | head -20
Nine times out of ten one file is most of the problem, and its name tells you which service to go and look at.
Add up du and it comes to almost nothing. df -h /var still says 100%. The space is held by a file that has been deleted while a process still has it open.
Unlinking a file removes the name. The data stays on disk until the last open file descriptor closes. du walks names, so it cannot see the file. df asks the filesystem, so it can.
Rotation is how servers get into this state, and it takes two steps.
First, logrotate renames app.log to app.log.1. A daemon that was never told to reopen its files keeps writing to the same inode, which now has a different name. Nothing is hidden yet. The file is visible, under a name that says it is old, growing.
Then, some cycles later, rotate 5 deletes the oldest copy. That is the inode the daemon is still writing to. The name goes. The descriptor does not. From here the file grows with no name at all, and every tool that works by listing directories is blind to it.
Find them:
sudo find /proc/[0-9]*/fd -ls 2>/dev/null | grep '(deleted)'
You get lines like this:
10256008 0 l-wx------ 1 root root 64 Sep 20 11:04 /proc/1234/fd/3 -> /var/log/app.log (deleted)
Process 1234 is holding file descriptor 3, and that descriptor points at a log file nobody can list any more.
lsof +L1 does the same job and is easier to read. It is also not installed on a minimal server, and a minimal server with a full disk is not the moment to start installing things. The find above needs nothing.
Do not reach for ls -l /proc/*/fd | grep '(deleted)'. It prints the matching descriptors and drops the /proc/<pid>/fd: headers that name the process, so you learn a file is held and not by what.
Restarting the process closes the descriptor and releases the space. That is the honest fix, and sometimes you cannot afford it.
You can truncate the file through its own descriptor instead:
sudo sh -c ': > /proc/1234/fd/3'
The space comes back immediately. The process keeps its descriptor and keeps writing. Measured on a 30 MB deleted-but-open file: du reported nothing before and after, and the bytes held by the descriptor went from 31457280 to 0.
One caveat. The process is still writing at its old offset, so the file stays sparse and its apparent size keeps climbing even though the blocks are gone. It buys you time to schedule a restart. It is not a permanent state to leave a server in.
A daemon reopens its log files when it is told to. Logrotate’s job is to tell it.
Look at the config for the file that grew:
grep -rl '/var/log/app' /etc/logrotate.conf /etc/logrotate.d/
Two ways to get the reopen right.
postrotate with a signal. The correct one for most daemons, because there is no window in which lines can be lost:
/var/log/nginx/*.log {
daily
rotate 14
missingok
compress
delaycompress
notifempty
sharedscripts
postrotate
/usr/sbin/nginx -s reopen
endscript
}
nginx -s reopen sends SIGUSR1, which is NGX_REOPEN_SIGNAL in nginx’s source. rsyslog takes SIGHUP. Check the daemon’s own documentation rather than guessing; the wrong signal to the right process reloads a config or kills it.
sharedscripts runs postrotate once for the whole pattern instead of once per matched file. Without it, a glob matching six files runs your reopen six times.
copytruncate. For a daemon that has no reopen signal at all. Logrotate copies the file and then truncates the original in place, so the descriptor stays valid and the inode never changes.
It is a fallback, not a default. The man page is explicit about the cost: “there is a very small time slice between copying the file and truncating it, so some logging data might be lost.” On a log that matters for billing or security, that is a real objection.
sudo logrotate -d /etc/logrotate.conf
-d makes no changes to the logs and does not update the state file. It only prints what it would do. Read it before you trust it.
To rotate now, once you believe the config:
sudo logrotate -f /etc/logrotate.conf
Two more things that quietly stop rotation.
Logrotate is not running. Distributions moved it from cron to a systemd timer at different times. Check both:
systemctl list-timers logrotate.timer
ls -l /etc/cron.daily/logrotate
The su directive is missing. Logrotate refuses to rotate a file whose parent directory is world-writable, or group-writable by a group other than root. It prints this and moves on:
error: skipping "/var/log/myapp/app.log" because parent directory has insecure
permissions (It's world writable or writable by group which is not "root") Set
"su" directive in config file to tell logrotate which user/group should be used
for rotation.
It is an error, the file is skipped, and nobody reads the output of a timer. So the log grows for months while logrotate runs on schedule and does nothing.
Tell it which user to rotate as:
/var/log/myapp/*.log {
su myapp myapp
weekly
rotate 8
}
Chowning the directory to root works too. Pick whichever matches how the application creates its files.
The journal. It has its own size caps and ignores logrotate entirely.
journalctl --disk-usage
See clearing journalctl logs and limiting journal disk usage for vacuuming it and for the caps that keep it vacuumed.
A web server’s access log. One busy site writes gigabytes a week. The access log grows quietly, because nothing in it looks like a problem. Reading Nginx access logs covers what is in an access line and where the file lives. On Apache, checking Apache error logs covers finding the log directory, which is the first step to measuring it.
Docker. The default json-file driver does not rotate anything unless you configure it, and the files do not live in /var/log at all:
sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head
See viewing Docker container logs for capping it in daemon.json.
A log level someone raised during an incident. Debug logging turned on at 2am and never turned off is a common cause, and the fix is not a rotation setting.
Everything above is maintenance on a design where each server stores its own logs on its own disk.
That design has one failure mode built in: the log grows until it takes the machine down, and it takes down every service on the machine, not just the one that was noisy. A full /var stops writes, and a daemon that cannot write its log often cannot start.
Moving the logs off the box removes the class of problem. The local file becomes a buffer with a small cap, and the copy that matters lives somewhere with its own retention policy.
Install the CL Agent and point it at your server. In /etc/clagent.toml:
URL = "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
Then cap the local journal hard, because the local copy is no longer the record. That is a different file, /etc/systemd/journald.conf:
[Journal]
SystemMaxUse=200M
Restart journald afterwards with sudo systemctl restart systemd-journald.
A disk alert is worth more than any of this, because the fix is easy at 85% and an outage at 100%.
Central Logging’s host monitoring collects disk usage from each host. Alerting on a disk running out of space covers the rule.
The other half is noticing when a host goes quiet, which is what a full disk often looks like from outside. Create a rule on the source, set it to Alert on empty, and use:
SELECT 1 FROM logs
WHERE timestamp > CAST(strftime('%s','now','-15 minutes') AS INTEGER)
LIMIT 1;
Alert-on-empty fires when the result set is empty. Here that means the host has sent nothing for fifteen minutes. Rules run every five minutes.
💌 Get notified on new features and updates