Guides

Step-by-step guides for getting logs off your servers and into one searchable place, then querying and alerting on them once they land.

Every guide covers three things: how to configure the source, how to ship the logs so a network blip does not lose them, and which queries are worth running on the result.

Running a homelab or a small rack? Start with Proxmox for the hypervisor and OPNsense, Synology and TrueNAS for the appliances that only speak syslog. Reading a Proxmox node’s logs where they sit? Checking Proxmox VE logs covers the journal, task logs, and why a failed scheduled backup never logs end task.

Logs already landing? Querying logs with SQL is where they start earning their keep.

No agent, just a script that prints things? Sending logs to a server with curl covers posting one line or a whole file over HTTP, and the four mistakes that make a failed upload look like a successful one.

Chasing a cron job instead? Cron job not running covers the file names, permissions and missing newline that make cron skip a job without a word. Checking cron logs covers where cron’s own log lives on each distribution and why it is so often empty, where cron job output goes covers keeping what a job printed, and monitoring cron jobs covers noticing when it stops printing anything at all.

Working with systemd? Viewing logs for a service with journalctl -u covers reading the journal, and saving journalctl output to a file covers getting it out of the journal without losing entries. journalctl printing nothing at all? “No journal files were found” covers the four different messages it uses to say that, and why three of them exit 0. A service that will not stay up? Why a systemd service failed to start reads systemctl status and journalctl -xeu, and decodes status=203/EXEC.

Narrowing the journal down? Viewing journalctl logs for a specific time range covers --since and --until and the midnight trap that quietly drops a day, showing only errors in journalctl covers -p err and why an application full of errors often returns nothing, and searching journalctl logs for a specific string covers -g, the case rule nobody expects, and the fields it cannot see. Only want the newest entries? Showing the last 100 lines in journalctl covers -n, why piping to grep gets the wrong 100, and why -g flips the order. Chasing the kernel? Viewing kernel logs with journalctl -k covers -k, why it hides everything before the last reboot even with --since, and why dmesg says “Operation not permitted”.

Tracking down a specific failure? Checking SSH logs on Linux covers auth.log, /var/log/secure and the journal, and why journalctl -t sshd finds no logins since OpenSSH 9.8. Checking failed SSH login attempts covers reading the auth log and counting attempts by source, checking Nginx error logs covers finding the file and raising the level when it looks empty, and viewing Docker container logs with docker logs covers tail, follow, grep and the three reasons it returns nothing. Need the file itself? Docker logs location covers where each driver keeps it, why LogPath is often empty, and saving logs to a file without losing stderr.

Reading a web server’s logs? Reading Nginx access logs covers the combined format field by field, the awk mistake that counts a 200 as a server error, and why half your requests can be missing from the file. Checking Apache error logs covers finding the file on each distribution, what LogLevel warn throws away, and the three places combined means three different things.

Parsing the journal, or reading one that came off another machine? Getting JSON output from journalctl covers -o json and the three cases where a value is not a string, including the one that turns your longest messages into null. Reading journal files from another server covers --file, -D and --root, and the four reasons a copied journal looks empty when it is not. Reading only what is new since last time? Using a journalctl cursor covers --cursor-file and --after-cursor, and the six ways a cursor repeats or loses entries.

Disk full? Fixing /var/log filling up covers finding what is using the space, including the deleted files du cannot see, and the logrotate settings that let it happen. Clearing journalctl logs and limiting journal disk usage covers vacuuming and the caps that keep it vacuumed, limiting Docker log size covers the container logs Docker never rotates by default, and alerting on a disk running out of space covers hearing about the next one at 85% instead of 100%. Server rebooted and nobody knows why? Finding out why a Linux server rebooted covers reading the previous boot, and what to do when there is no previous boot to read.

Want to hear about problems instead of going to look for them? Alerting on errors in your logs covers writing rules that actually fire, including the one nobody writes — alert when the logs stop. Seeing who logged in over SSH covers reading successful logins with last and the journal, and the two files that hide half of them.

No log server yet? Start with setting up a self-hosted logging server, then come back and pick a source.