Guides

Step-by-step guides for getting logs off your servers and into one searchable place, then querying and alerting on them once they land.

Every guide covers three things: how to configure the source, how to ship the logs so a network blip does not lose them, and which queries are worth running on the result.

Running a homelab or a small rack? Start with Proxmox for the hypervisor and OPNsense, Synology and TrueNAS for the appliances that only speak syslog.

Logs already landing? Querying logs with SQL is where they start earning their keep.

Chasing a cron job instead? Checking cron logs covers where cron’s own log lives on each distribution and why it is so often empty, where cron job output goes covers keeping what a job printed, and monitoring cron jobs covers noticing when it stops printing anything at all.

Working with systemd? Viewing logs for a service covers reading the journal, and saving journalctl output to a file covers getting it out of the journal without losing entries.

Narrowing the journal down? Viewing journalctl logs for a specific time range covers --since and --until and the midnight trap that quietly drops a day, showing only errors in journalctl covers -p err and why an application full of errors often returns nothing, and searching journalctl logs for a specific string covers -g, the case rule nobody expects, and the fields it cannot see.

Tracking down a specific failure? Checking failed SSH login attempts covers reading the auth log and counting attempts by source, checking Nginx error logs covers finding the file and raising the level when it looks empty, and viewing Docker container logs covers docker logs and the three reasons it returns nothing.

Parsing the journal, or reading one that came off another machine? Getting JSON output from journalctl covers -o json and the three cases where a value is not a string, including the one that turns your longest messages into null. Reading journal files from another server covers --file, -D and --root, and the four reasons a copied journal looks empty when it is not.

Journal eating the disk? Clearing journalctl logs and limiting journal disk usage covers vacuuming and the caps that keep it vacuumed, and alerting on a disk running out of space covers hearing about the next one at 85% instead of 100%. Server rebooted and nobody knows why? Finding out why a Linux server rebooted covers reading the previous boot, and what to do when there is no previous boot to read.

Want to hear about problems instead of going to look for them? Alerting on errors in your logs covers writing rules that actually fire, including the one nobody writes — alert when the logs stop. Alerting on SSH logins applies it to the auth log.

No log server yet? Start with setting up a self-hosted logging server, then come back and pick a source.