Sending Proxmox logs to a central server

Introduction

Proxmox keeps each task log on the node that ran the task. Lose the node, lose the evidence.

That is fine with one server. It stops being fine the moment you own three. It stops being fine the moment a node reboots at 3am and the reason is on a disk you cannot reach.

This guide moves every Proxmox log into one searchable place:

1. Node logs — pvedaemon, pveproxy, pvestatd, corosync, the kernel, ZFS.
2. Task logs — backups, migrations, snapshots, and whether they finished.
3. Guest logs — the VMs and containers themselves.

It ends with an alert that pages you when a backup fails, which is the reason most people start.

What Proxmox logs, and where it puts them

Proxmox VE is Debian with systemd underneath. That means most of it is already in the journal:

The split matters. The journal tells you a backup ran and whether it succeeded. The file under /var/log/pve/tasks/ holds the several hundred lines that say why it failed.

Ship the journal first. It covers most of what you want and it takes five minutes.

Step 1: Install the agent on each node

The CL Agent reads the journal with journalctl -o json and posts new entries upstream. It tracks a cursor, so it never sends the same line twice.

Create a log source per node in Central Logging. One source per node keeps pve1 and pve2 apart, and you can still search across all of them at once.

On each node:

wget https://downloads.eligian.com/clagent-linux-amd64.tar.gz
tar xzf clagent-linux-amd64.tar.gz
sudo mv clagent-linux-amd64 /usr/local/bin/clagent

Write /etc/clagent.toml with the URL for that node’s source:

URL = "https://logs.example.com/your-log-source-token"

Install and start it:

sudo /usr/local/bin/clagent -install
sudo systemctl enable clagent
sudo systemctl start clagent

Within a minute the node’s journal starts arriving. You also get host monitoring for free — the agent checks in on a schedule, and Central Logging alerts you if a node goes quiet. On a cluster that is worth having on its own.

Step 2: Find your task lines

Proxmox writes a start and an end line to the journal for every task. Check the exact wording on your version before you build anything on top of it:

journalctl -u pvedaemon --since "24 hours ago" | grep "end task"

You should see one line per finished task. Successful tasks end in OK. Failed tasks end with the error instead.

That single fact is the whole backup-alerting story. You do not need the task output files to know something broke. You need them to know why.

Step 3: Ship the task output files as well

Skip this step if the journal is enough for you. Add it when you want to read a failed backup’s full output without SSHing into the node.

Proxmox appends one line per finished task to /var/log/pve/tasks/index. Ship it with an offset script so you only send what is new:

#!/bin/bash
# /usr/local/bin/ship-pve-tasks.sh
set -euo pipefail

LOG=/var/log/pve/tasks/index
OFFSET_FILE=/var/lib/cl-shipper/pve-tasks.offset
ENDPOINT="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

mkdir -p "$(dirname "$OFFSET_FILE")"
offset=$(cat "$OFFSET_FILE" 2>/dev/null || echo 0)
size=$(stat -c %s "$LOG")

# Proxmox rotates the index; start over when it shrinks.
if [ "$size" -lt "$offset" ]; then
    offset=0
fi

if [ "$size" -eq "$offset" ]; then
    exit 0
fi

tail -c "+$((offset + 1))" "$LOG" \
    | curl -sf -X POST "$ENDPOINT" --data-binary @-

echo "$size" > "$OFFSET_FILE"

Run it every minute:

* * * * * /usr/local/bin/ship-pve-tasks.sh

set -euo pipefail earns its place here. The offset only advances after a successful POST, so a failed upload retries next minute instead of disappearing.

Those index lines are Proxmox’s internal UPID format, not JSON. Search them with full-text search rather than SQL.

Look in /var/log/pve/ on your own node before you go further. Different Proxmox versions put different things there, and the same script points at any of them — change LOG and the offset file name.

Step 4: Ship the guests

A VM is a Linux box. Treat it like one. Install the agent inside it with its own log source and it behaves exactly like a bare-metal server.

Containers are the same, with one wrinkle. An unprivileged LXC container runs its own journal, and the host cannot read it. So either install the agent inside the container too, or have the application post to the ingest endpoint directly:

curl -X POST https://logs.example.com/api/v1/log/YOUR-SOURCE-TOKEN \
    -d '{"level":"error","service":"invoicer","msg":"stripe webhook rejected"}'

For a container running one small service, the second option is less to maintain than an agent.

Step 5: Query what arrives

The agent sends journalctl -o json output, so each row is a journal entry with the human-readable text in MESSAGE. Central Logging stores rows in SQLite, so json_extract reaches any field.

Every task that did not end in OK, across every node:

SELECT
    datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000,
             'unixepoch')                  AS happened,
    json_extract(msg, '$._HOSTNAME')       AS node,
    json_extract(msg, '$.MESSAGE')         AS task
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.MESSAGE') LIKE '%end task%'
  AND json_extract(msg, '$.MESSAGE') NOT LIKE '%OK'
ORDER BY "timestamp" DESC
LIMIT 100;

Two timestamps appear in that query and they are not the same thing. The timestamp column is when Central Logging received the line. __REALTIME_TIMESTAMP is when the event happened on the node, in microseconds. They are usually within a minute of each other. After a network outage they are not, so sort by the one you mean.

Storage and ZFS problems, which is where a homelab node usually dies:

SELECT
    "timestamp",
    json_extract(msg, '$._HOSTNAME')          AS node,
    json_extract(msg, '$.SYSLOG_IDENTIFIER')  AS unit,
    json_extract(msg, '$.MESSAGE')            AS message
FROM logs
WHERE json_valid(msg)
  AND (json_extract(msg, '$.MESSAGE') LIKE '%ZFS%'
       OR json_extract(msg, '$.MESSAGE') LIKE '%I/O error%'
       OR json_extract(msg, '$.MESSAGE') LIKE '%degraded%')
ORDER BY "timestamp" DESC
LIMIT 200;

Who logged into the web interface:

SELECT
    "timestamp",
    json_extract(msg, '$._HOSTNAME') AS node,
    json_extract(msg, '$.MESSAGE')   AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'pvedaemon'
  AND json_extract(msg, '$.MESSAGE') LIKE '%authentication failure%'
ORDER BY "timestamp" DESC
LIMIT 100;

Point that last one at an internet-facing node and you will find out how much of the internet is trying root@pam.

Step 6: Alert on failed backups

A query you have to remember to run is a query you will not run. Turn this one into an alert rule:

SELECT COUNT(*) AS failures
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.MESSAGE') LIKE '%end task%'
  AND json_extract(msg, '$.MESSAGE') NOT LIKE '%OK'
HAVING failures > 0;

Central Logging evaluates rules every five minutes and notifies through Slack or Telegram. Set a max frequency on the rule. A node with a dying disk will otherwise fail the same task forty times before breakfast.

Bonus: catch the backup that never starts

The alert above fires when a backup fails. It says nothing when a backup never runs, and a backup job that silently stopped scheduling is worse than one that fails loudly.

Cron monitoring covers that gap. vzdump calls a hook script at each phase of a backup job, so have it ping Central Logging:

#!/bin/bash
# /usr/local/bin/cl-backup-hook.sh
PHASE="$1"
PING="https://logs.example.com/p/YOUR-API-KEY/proxmox-backup"

case "$PHASE" in
  job-start) curl -fsS "$PING?state=start" >/dev/null || true ;;
  job-end)   curl -fsS "$PING?state=complete" >/dev/null || true ;;
  job-abort) curl -fsS "$PING?state=fail&message=vzdump+job+aborted" >/dev/null || true ;;
esac

exit 0

Make it executable and register it in /etc/vzdump.conf:

script: /usr/local/bin/cl-backup-hook.sh

Note the || true and the exit 0. A hook script that exits non-zero aborts the backup. Do not let a hiccup reaching your log server destroy the thing you were trying to protect.

Central Logging now knows how long the job should take and alerts you when a start never gets its complete.

Summary

Install the agent on each node for the journal. Add an offset script if you want the full task output. Treat guests as separate machines. Then let SQL and alert rules do the watching.

Central Logging is a single binary you install on a server you already own. It stores logs in SQLite, needs no JVM and no Docker, and costs $187 once. See Deploy to get an instance running, or start with setting up a self-hosted logging server.