How to get alerted when errors appear in your logs

The alert you never got

A log alert that never fires looks exactly like a system with no problems.

That is the failure mode worth designing against. A broken rule does not announce itself. It sits in the list marked active, matches nothing for eight months, and you find the outage from a customer email instead.

Most of this guide is about making rules that actually match. The rest is about what happens after one does.

What an alert rule is

A query and a condition.

Central Logging runs your query against the log source every five minutes. It does not read the rows that come back. It checks one thing: was the result empty or not?

That single fact shapes every rule you write.

• Alert on results — the query found something. Errors appeared.
• Alert on empty — the query found nothing. Something that should be there is missing.

Because only emptiness matters, the columns you select are irrelevant. SELECT 1 ... LIMIT 1 is the whole shape of a good alert query. Selecting fifty columns and a thousand rows costs you time and buys nothing.

Start with full-text search

The simplest rule needs no SQL. An FTS rule takes a search term and matches it against the raw log line:

"Failed password"

Quotes make it a phrase. Without them you match every line containing either word.

FTS handles the boolean operators too:

error AND payment
timeout OR "connection refused"

Full-text search matches the line as stored. If you ship logs with the CL Agent, that stored line is a journald JSON object, so the field names are searchable alongside the values. Searching sshd matches lines where sshd appears anywhere, including in SYSLOG_IDENTIFIER.

Use FTS when a keyword is enough. Reach for SQL when it is not.

Move to SQL when you need structure

SQL rules run against the log source’s own database. The table is logs:

Your application writes JSON? Match on a field:

SELECT 1 FROM logs
WHERE timestamp > CAST(strftime('%s','now','-10 minutes') AS INTEGER)
  AND json_valid(msg)
  AND json_extract(msg, '$.level') IN ('error','fatal')
LIMIT 1

Pair that with alert on results.

Note timestamp is arrival time, not event time. Nothing parses a timestamp out of your log line. For alerting that is the behaviour you want — you are asking “did something break in the last ten minutes”, and arrival is what you can act on. For investigating afterwards, extract the event’s own timestamp instead.

Logs shipped by the CL Agent nest one level deeper

The agent sends journalctl -o json, so every row is a journal entry. Your application’s own JSON sits inside MESSAGE as a string. Reaching a field takes two hops:

SELECT 1 FROM logs
WHERE timestamp > CAST(strftime('%s','now','-10 minutes') AS INTEGER)
  AND json_valid(msg)
  AND json_valid(json_extract(msg, '$.MESSAGE'))
  AND json_extract(json_extract(msg, '$.MESSAGE'), '$.level') IN ('error','fatal')
LIMIT 1

Write the single-hop version against agent-shipped logs and you get a rule that matches nothing, forever.

Alert when the logs stop

This is the rule most people never write, and it is the one that catches real outages.

Every alert above fires on something bad appearing. Invert it. Alert when something good stops appearing:

SELECT 1 FROM logs
WHERE timestamp > CAST(strftime('%s','now','-1 hour') AS INTEGER)
LIMIT 1

Set the condition to alert on empty.

Now you hear about it when a source goes quiet. The agent died, the server rebooted into a broken state, a firewall rule changed, the disk filled and journald stopped writing. All of those look identical from your side: nothing arrives. None of them trigger an error-matching rule, because errors are logs too, and no logs are arriving.

Narrow it to a specific heartbeat when the source is chatty and you want to know a particular job still runs:

SELECT 1 FROM logs
WHERE timestamp > CAST(strftime('%s','now','-2 hours') AS INTEGER)
  AND msg LIKE '%nightly-sync finished%'
LIMIT 1

Pick a window comfortably longer than the real interval. A job that runs hourly and occasionally takes forty minutes needs a two-hour window, or you will alert on a slow night rather than a broken one.

For cron jobs specifically, cron monitoring does this properly — it learns the schedule instead of making you guess a window.

Thresholds need a subquery

One error is noise. Fifty errors in ten minutes is an incident. Counting sounds like a job for HAVING:

-- This does not work.
SELECT 1 FROM logs
WHERE json_valid(msg) AND json_extract(msg, '$.level') = 'error'
HAVING COUNT(*) >= 50

SQLite rejects it: HAVING clause on a non-aggregate query. The rule errors out on every evaluation and never fires.

Put the count in a scalar subquery instead:

SELECT 1 WHERE (
  SELECT COUNT(*) FROM logs
  WHERE timestamp > CAST(strftime('%s','now','-10 minutes') AS INTEGER)
    AND json_valid(msg)
    AND json_extract(msg, '$.level') = 'error'
) >= 50

Below the threshold the query returns no rows. At or above it returns one. That is exactly the empty-or-not signal the alert engine reads.

Four ways to write a rule that never fires

Every one of these is silent. The rule looks active and matches nothing.

1. json_extract meets a line that is not JSON. One plain-text line in the source is enough. json_extract raises “malformed JSON”, the evaluation aborts, and the failure goes to the server log where you are not looking. Guard it:

WHERE json_valid(msg) AND json_extract(msg, '$.level') = 'error'

Order matters. SQLite short-circuits AND, so json_valid(msg) protects the terms after it. It does not protect expressions in the SELECT list — but for an alert you are selecting 1, so this stays simple.

2. A string compared as a number. journald writes PRIORITY as a string. This matches nothing, and reports no error:

-- Silently matches zero rows.
WHERE json_valid(msg) AND json_extract(msg, '$.PRIORITY') <= 3

-- Correct.
WHERE json_valid(msg) AND CAST(json_extract(msg, '$.PRIORITY') AS INTEGER) <= 3

The same applies whenever you compare an extracted value against strftime('%s', ...). Cast both sides. Comparing against the timestamp column works without a cast because the column is already an integer, but casting costs nothing and removes the question.

3. The window is tighter than the evaluation interval. Rules run every five minutes. A rule looking back exactly five minutes has no slack, and a line landing near the boundary falls between two runs. Look back ten minutes for a five-minute cadence. Alerting twice about the same error is cheaper than missing it.

4. The nesting level is wrong. Covered above, and it is the most common one. Search the raw line first to see what actually arrived, then write the extract to match.

Test every rule before you trust it. Run the query on the search page and confirm it returns rows right now, while you know the condition is true. A rule you have never seen match is a rule you have no reason to believe in.

Where the notification goes

Central Logging sends alerts to one notification channel, not to all of them.

Tick Send alerts to this channel on the channel you want. Only one channel holds it, so ticking it elsewhere clears it here. With no channel marked, alerts go to the first active channel by creation order.

Check it rather than assume it. Each channel has its own Test button, and the test sends to the channel you clicked — so testing Telegram proves Telegram works, not that alerts go there. The Alerts page labels the channel that receives them “Alerts go here”.

How often you will hear about it

Each rule has a Max Frequency: the minimum gap between two notifications from that rule. The choices are 5 minutes, 15 minutes, 1 hour, 6 hours and once a day. New rules default to once a day.

Nothing shorter than 5 minutes is offered, because rules are only evaluated every 5 minutes.

Set it by how you would respond. A failed payment webhook is worth a nudge every 15 minutes until someone fixes it. A flood of application errors is not — one message a day beats ninety-six about the same broken thing.

Two more behaviours worth knowing:

• Notifications go out within a minute of a rule firing. The five-minute evaluation is the slow part, not the delivery.
• If more than ten alerts are queued unsent, the excess is dropped so the queue cannot jam. In practice you hit this only when a notification channel is misconfigured and nothing is draining.

When the condition clears, the rule stops showing as firing within about eight minutes. No all-clear message is sent. Alerts tell you something started. They do not tell you it stopped.

What is worth alerting on

Alert on things you would act on tonight. Everything else belongs in a dashboard or a weekly look at the logs.

1. A source going quiet. The alert-on-empty rule above. Highest value, least written.
2. Error rate crossing a threshold. Not single errors. A rate.
3. Anything touching money. Failed payments, failed webhook deliveries, failed invoicing.
4. Authentication anomalies. Root logins, failure bursts — see alerting on SSH logins.
5. Certificate and disk warnings. Slow-moving failures nobody watches until they land.

Resist alerting on every ERROR line. A rule that fires daily gets muted in a week, and a muted rule is worse than no rule — it costs the same and tells you nothing.

Summary

Write the query so it returns rows only when you want to be woken up. Select 1, limit to 1, and let the emptiness carry the signal.

Guard every json_extract with json_valid. Cast anything journald stores as a string. Use a subquery for thresholds, not HAVING. Give the window more slack than the five-minute evaluation.

Then write the rule nobody writes: alert when the logs stop.

Central Logging runs log search, alerting, uptime checks and cron monitoring from one binary on your own server. See Alerting for the reference, or Deploy to get started.