How to get an alert when a Linux server is running out of disk space

Introduction

A full disk never announces itself. The database announces it, at 3am, by refusing to write.

The warning was available for days. Nothing was watching. This guide sets up the watching: a small cron job that reports df output as a log line, a rule that fires while there is still room to act, and the two mistakes that make such a rule fire on the wrong things or on nothing at all.

What Central Logging does and does not collect

Worth saying before you build on it. Central Logging has no disk collector. Host monitoring tracks check-in liveness and inventory — hostname, OS, kernel, architecture, external IP, uptime, processes — and alerts when a host stops checking in. It is not CPU or disk threshold monitoring.

Metric sources scrape the Prometheus text format and chart Counters and Gauges. They chart. They do not alert.

What does alert is a SQL rule over your logs. So the working shape is: make the disk usage be a log line, then alert on the line. That takes about five minutes and it has an advantage over a dedicated agent — the same rule engine, the same notification channel, and the same query language you already use for everything else.

If you want graphs of disk usage over time as well, run node_exporter and point a metric source at it. Alerting still comes from the log line.

Report the disk usage

Create /usr/local/bin/dfreport:

#!/bin/sh
# Report disk usage to Central Logging as one JSON line per mount.
TOKEN=YOUR-SOURCE-TOKEN
URL="https://logs.example.com/api/v1/ingest_logs/$TOKEN"

df -P -x tmpfs -x devtmpfs -x overlay -x squashfs | awk -v h="$(hostname)" '
  NR > 1 {
    gsub(/%/, "", $5)
    printf "{\"check\":\"disk\",\"host\":\"%s\",\"mount\":\"%s\",\"used_pct\":%d,\"avail_kb\":%d}\n", h, $6, $5, $4
  }' | curl -sS --data-binary @- "$URL"

chmod +x /usr/local/bin/dfreport, then run it once. You should get {"logs_ingested": 3} or similar back, and the lines should appear in the log view:

{"check":"disk","host":"web1","mount":"/","used_pct":91,"avail_kb":1258291}
{"check":"disk","host":"web1","mount":"/var","used_pct":44,"avail_kb":18874368}

Four details in that script earn their place.

df -P. POSIX output format. Without it, a long device name wraps onto a second line and awk reads a field that is not there.

The -x flags. Every container overlay and tmpfs is at some alarming percentage and none of them matter. Excluding them up front is easier than excluding them in SQL later.

used_pct is written as a number, not a string. %d in the printf, no quotes around it in the JSON. This matters more than it looks — see below.

--data-binary @-, not -d @-. curl -d strips newlines. The bulk endpoint splits on newlines. Use -d and your three mounts arrive as one concatenated line that is not valid JSON, so every rule below silently skips it. There is no error. The row is just quietly useless.

Install it in cron:

*/15 * * * * /usr/local/bin/dfreport >/dev/null 2>&1

Fifteen minutes is a sensible cadence. Disks fill over hours, and the rule engine only evaluates every five minutes anyway.

Getting the token: create a log source in Central Logging and copy its token, or reuse the source your other logs already go to. Details in sending logs.

The alert rule

Create an alert rule with condition Alert on results and this query:

SELECT 1 FROM logs
WHERE json_valid(msg)
  AND json_extract(msg,'$.check') = 'disk'
  AND CAST(json_extract(msg,'$.used_pct') AS INTEGER) >= 85
  AND "timestamp" > CAST(strftime('%s','now','-30 minutes') AS INTEGER)
LIMIT 1;

That is the whole thing. Rules run every five minutes, and the only thing that matters is whether the result set came back empty, so SELECT 1 ... LIMIT 1 is the right shape. Selecting columns is wasted work.

Three parts are load-bearing.

json_valid(msg) first. One plain-text line anywhere in the source and json_extract raises “malformed JSON” for the whole query. The guard belongs in the WHERE clause, where SQLite short-circuits AND and the extracts after it never run on a bad row. In a SELECT list it protects nothing.

The 30-minute window. Without it the rule matches the report from last Tuesday and fires forever. With a 15-minute cron and a 30-minute window you tolerate one missed run.

CAST(... AS INTEGER). The next section is about that one.

The CAST that stops a 3% disk paging you

Drop the CAST and the rule still looks correct. It is not.

SQLite sorts every TEXT value above every INTEGER, whatever the text says. So if used_pct ever arrives as a string, this happens:

SELECT '3' >= 85;     -- 1
SELECT '96' >= 85;    -- 1
SELECT 'abc' >= 85;   -- 1

A disk at 3% fires your disk-full alert. So does a mount whose percentage came through as a word.

It is easy to end up with strings. Any pipeline that builds the JSON with %s instead of %d, or with jq -R, or by hand in a language that stringifies numbers, produces "used_pct":"91". The rule keeps working, and then one day it fires at 3%.

CAST(json_extract(msg,'$.used_pct') AS INTEGER) converts the text before comparing, and '3' becomes 3. It costs nothing on rows that were already numbers.

The same trap runs the other way for journald fields. PRIORITY is stored as a string, so json_extract(msg,'$.PRIORITY') <= 3 matches zero rows and raises no error. Cast both sides of any comparison against an extracted value and the question goes away.

Per-mount thresholds

One threshold across every mount produces noise. A /boot partition sits at 85% forever and is fine. A database volume at 85% is not.

SELECT 1 FROM logs
WHERE json_valid(msg)
  AND json_extract(msg,'$.check') = 'disk'
  AND "timestamp" > CAST(strftime('%s','now','-30 minutes') AS INTEGER)
  AND CAST(json_extract(msg,'$.used_pct') AS INTEGER) >=
      CASE json_extract(msg,'$.mount')
        WHEN '/boot' THEN 95
        WHEN '/var/lib/postgresql' THEN 80
        ELSE 90
      END
LIMIT 1;

Percentages mislead in one more way. 10% free on a 200GB volume is 20GB and a week of breathing room. 10% free on a 20GB volume is two hours. That is what avail_kb is in the log line for:

SELECT 1 FROM logs
WHERE json_valid(msg)
  AND json_extract(msg,'$.check') = 'disk'
  AND CAST(json_extract(msg,'$.avail_kb') AS INTEGER) < 2097152
  AND "timestamp" > CAST(strftime('%s','now','-30 minutes') AS INTEGER)
LIMIT 1;

Under 2GB free, whatever the percentage says. Run both rules. They catch different disks.

Alert when the report stops arriving

A rule that fires on high usage cannot fire if nothing is reporting. The host that fell over is the one you hear nothing from.

Set the condition to Alert on empty and invert the question:

SELECT 1 FROM logs
WHERE json_valid(msg)
  AND json_extract(msg,'$.check') = 'disk'
  AND json_extract(msg,'$.host') = 'web1'
  AND "timestamp" > CAST(strftime('%s','now','-45 minutes') AS INTEGER)
LIMIT 1;

Empty means web1 has not reported in 45 minutes. One rule per host, which is tedious past a handful of machines — at that point use host monitoring instead, which does exactly this for every host with the CL Agent installed and no rules to write.

If you ship through the CL Agent

Already shipping the journal with the CL Agent? Send the report through the journal instead of curling it, and you have one shipping path rather than two.

Cron’s stdout does not reach the journal on its own — it goes to mail. Pipe it to logger:

df -P -x tmpfs -x devtmpfs -x overlay -x squashfs | awk -v h="$(hostname)" '
  NR > 1 {
    gsub(/%/, "", $5)
    printf "{\"check\":\"disk\",\"host\":\"%s\",\"mount\":\"%s\",\"used_pct\":%d,\"avail_kb\":%d}\n", h, $6, $5, $4
  }' | logger -t dfreport

It costs one extra hop in every query. The agent sends journalctl -o json, so each row is a journal entry object and your JSON sits inside MESSAGE as a string:

SELECT 1 FROM logs
WHERE json_valid(msg)
  AND json_valid(json_extract(msg,'$.MESSAGE'))
  AND json_extract(json_extract(msg,'$.MESSAGE'),'$.check') = 'disk'
  AND CAST(json_extract(json_extract(msg,'$.MESSAGE'),'$.used_pct') AS INTEGER) >= 85
  AND "timestamp" > CAST(strftime('%s','now','-30 minutes') AS INTEGER)
LIMIT 1;

Two json_valid guards, because the nested string can be malformed independently of the row around it.

Curling directly is simpler and keeps the rules readable. Go through the journal when you would rather maintain one shipping path, and because the reading is then also on the host itself, where journalctl -t dfreport can show it after the fact.

The curl route does not retry. A POST that fails drops that reading, and nothing goes back for it. That is tolerable here only because a df reading is cheap and repeats in fifteen minutes. Do not reason the same way about application logs.

The journal route does retry. The agent moves its cursor only after your instance has accepted the entries, so a reading it could not deliver goes out again on the next pass.

Charting it as well

Send the report as a cron monitoring ping and each number becomes a chart on the job page, over the last hundred runs:

curl -fsS "https://logs.example.com/p/YOUR-API-KEY/disk-report?state=complete&metric=root_pct:91&metric=var_pct:44"

Metrics are parsed at the last colon, so names may contain colons. Twenty metrics per ping, 64-byte names. A malformed metric is dropped rather than rejected — a ping is the tail of a shell command, and failing it over a typo would cause the outage it is meant to report.

Charts are for the trend: which volume has been climbing all month, and whether last week’s cleanup held. Alerts still come from the log rule. Ping metrics do not have thresholds.

Where the alert goes

Alerts reach one notification channel — the one marked default on the Alerts page. Slack webhook, Telegram or Pushover. The rule engine runs every five minutes and delivery lands within a minute of a rule firing, so worst case is about six minutes from the df reading to your phone.

Max Frequency on the rule caps the repeats, and it defaults to 24 hours. A disk over 85% stays over 85%, so without that cap you would hear about it every five minutes. Shorten it to 1h or 6h when you want the reminder to nag. The pickable values are 5m, 15m, 1h, 6h and 24h — nothing below 5m, because that is how often rules run.

No all-clear message is sent when the disk drains. Treat an alert as “go look”.

What this does not catch

Honest limits, so you know what you are still exposed to.

A disk that fills in one step. A runaway log file or a bad dd can take a volume from 40% to 100% inside your 15-minute gap. Sampling cannot see between samples.

Inodes. df -i is a separate number, and a filesystem can be out of inodes at 12% used. Add a second awk block if your workload creates many small files.

The disk that is already full. dfreport writes nothing to disk, so it survives — but a cron job that cannot write its own temp file may not. Alert on the report stopping, not only on the number being high.

Where to go next

💌 Get notified on new features and updates

Only sent when a new version is released. Nothing else.