Sending Proxmox logs to a central server

Introduction

Proxmox keeps each task log on the node that ran the task. Lose the node, lose the evidence.

That is fine with one server. It stops being fine the moment you own three. It stops being fine the moment a node reboots at 3am and the reason is on a disk you cannot reach.

This guide moves every Proxmox log into one searchable place:

1. Node logs — pvedaemon, pveproxy, pvestatd, corosync, the kernel, ZFS.
2. Task logs — backups, migrations, snapshots, and whether they finished.
3. Guest logs — the VMs and containers themselves.

It ends with an alert that pages you when a backup fails, which is the reason most people start.

What Proxmox logs, and where it puts them

Proxmox VE is Debian with systemd underneath. That means most of it is already in the journal:

The split matters. The journal tells you a backup failed, in one line. The file under /var/log/pve/tasks/ holds the several hundred lines that say why.

Checking Proxmox VE logs covers reading each of these on the node itself.

Ship the journal first. It covers most of what you want and it takes five minutes.

Step 1: Install the agent on each node

The CL Agent reads the journal with journalctl -o json and posts new entries upstream. It tracks a cursor and moves it only once your instance has accepted the entries, so a node that loses the network for an hour ships the hour when it comes back.

Create a log source per node in Central Logging. One source per node keeps pve1 and pve2 apart, and you can still search across all of them at once.

On each node:

wget https://downloads.eligian.com/clagent-linux-amd64.tar.gz
tar xzf clagent-linux-amd64.tar.gz
sudo mv clagent-linux-amd64 /usr/local/bin/clagent

Write /etc/clagent.toml with the URL for that node’s source:

URL = "https://logs.example.com/your-log-source-token"

Install and start it:

sudo /usr/local/bin/clagent -install
sudo systemctl enable clagent
sudo systemctl start clagent

Within a minute the node’s journal starts arriving. You also get host monitoring for free — the agent checks in on a schedule, and Central Logging alerts you if a node goes quiet. On a cluster that is worth having on its own.

Step 2: Find your backup result lines

A task started from the web UI logs a start line and an end line, both from pvedaemon. The end line carries the result: OK, WARNINGS: 2, or the error text.

Scheduled backups do not log an end line. pvescheduler starts the backup task from a short-lived child process, and that child exits before the backup finishes. Only the process that started a task logs its end, so nothing does. A search for end task misses every failed nightly backup.

vzdump logs its own result lines, and those arrive whoever started the job:

journalctl --since "24 hours ago" -g 'Backup of VM .* failed|Backup job (failed|finished with errors)'

A failed guest looks like ERROR: Backup of VM 101 failed - <the reason>. A job with failures ends with INFO: Backup job finished with errors. Those two lines are the whole backup-alerting story. You need the task output to find out why. You do not need it to find out that something broke.

Step 3: Ship the task output files as well

Skip this step if the journal is enough for you. Add it when you want to read a failed backup’s full output without SSHing into the node.

Proxmox appends one line per finished task to /var/log/pve/tasks/index: the UPID, the end time in hex, and the status. Once the file passes 50,000 bytes, Proxmox renames it to index.1 and starts a new one. So the script tracks the file’s inode as well as its offset, and finishes the old file before it starts the new one:

#!/bin/bash
# /usr/local/bin/ship-pve-tasks.sh
set -euo pipefail

LOG=/var/log/pve/tasks/index
STATE=/var/lib/cl-shipper/pve-tasks.state
ENDPOINT="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

mkdir -p "$(dirname "$STATE")"
read -r last_inode offset 2>/dev/null < "$STATE" || { last_inode=0; offset=0; }

inode=$(stat -c %i "$LOG")

# Proxmox renamed the old index to index.1. Send the rest of it first.
if [ "$inode" != "$last_inode" ]; then
    if [ "$(stat -c %i "$LOG.1" 2>/dev/null)" = "$last_inode" ]; then
        old_size=$(stat -c %s "$LOG.1")
        if [ "$old_size" -gt "$offset" ]; then
            tail -c "+$((offset + 1))" "$LOG.1" | head -c "$((old_size - offset))" \
                | curl -sf -X POST "$ENDPOINT" --data-binary @- >/dev/null
        fi
    fi
    offset=0
    echo "$inode 0" > "$STATE"
fi

size=$(stat -c %s "$LOG")
if [ "$size" -gt "$offset" ]; then
    tail -c "+$((offset + 1))" "$LOG" | head -c "$((size - offset))" \
        | curl -sf -X POST "$ENDPOINT" --data-binary @- >/dev/null
fi

echo "$inode $size" > "$STATE"

Run it every minute:

* * * * * /usr/local/bin/ship-pve-tasks.sh

set -euo pipefail earns its place here. The state file only moves after a successful POST, so a failed upload retries next minute instead of disappearing. head -c stops at the size the script measured, so a line Proxmox appends mid-upload waits for the next run instead of being half-sent.

Leave /var/log/pve/tasks/active alone. Proxmox rewrites that file on every change, so an offset into it means nothing.

The index lines are Proxmox’s UPID format, not JSON. Search them with full-text search rather than SQL.

Step 4: Ship the guests

A VM is a Linux box. Treat it like one. Install the agent inside it with its own log source and it behaves exactly like a bare-metal server.

Containers are the same, with one wrinkle. An LXC container runs its own journal, and the host’s journal does not collect it. Root on the host can still read it with pct exec 101 -- journalctl, but an agent on the host will not ship it. So either install the agent inside the container too, or have the application post to the ingest endpoint directly:

curl -X POST https://logs.example.com/api/v1/log/YOUR-SOURCE-TOKEN \
    -d '{"level":"error","service":"invoicer","msg":"stripe webhook rejected"}'

For a container running one small service, the second option is less to maintain than an agent.

Step 5: Query what arrives

The agent sends journalctl -o json output, so each row is a journal entry with the human-readable text in MESSAGE. Central Logging stores rows in SQLite, so json_extract reaches any field.

Every failed task and failed backup, across every node:

SELECT
    datetime(CAST(json_extract(msg, '$.__REALTIME_TIMESTAMP') AS INTEGER) / 1000000,
             'unixepoch')                  AS happened,
    json_extract(msg, '$._HOSTNAME')       AS node,
    json_extract(msg, '$.MESSAGE')         AS task
FROM logs
WHERE json_valid(msg)
  AND (
        (json_extract(msg, '$.MESSAGE') LIKE '%end task UPID:%'
         AND json_extract(msg, '$.MESSAGE') NOT LIKE '%: OK'
         AND json_extract(msg, '$.MESSAGE') NOT LIKE '%: WARNINGS: %')
     OR json_extract(msg, '$.MESSAGE') LIKE 'ERROR: Backup of VM % failed%'
     OR json_extract(msg, '$.MESSAGE') LIKE 'ERROR: Backup job failed%'
     OR json_extract(msg, '$.MESSAGE') = 'INFO: Backup job finished with errors'
  )
ORDER BY "timestamp" DESC
LIMIT 100;

The end task half catches failed tasks started from the web UI: migrations, snapshots, restores, manual backups. It skips tasks that ended with warnings, which are not failures. The vzdump half catches backups, scheduled or not. A failed manual backup shows up in both halves.

Two timestamps appear in that query and they are not the same thing. The timestamp column is when Central Logging received the line. __REALTIME_TIMESTAMP is when the event happened on the node, in microseconds. They are usually within a minute of each other. After a network outage they are not, so sort by the one you mean.

Storage and ZFS problems, which is where a homelab node usually dies:

SELECT
    "timestamp",
    json_extract(msg, '$._HOSTNAME')          AS node,
    json_extract(msg, '$.SYSLOG_IDENTIFIER')  AS unit,
    json_extract(msg, '$.MESSAGE')            AS message
FROM logs
WHERE json_valid(msg)
  AND (json_extract(msg, '$.MESSAGE') LIKE '%ZFS%'
       OR json_extract(msg, '$.MESSAGE') LIKE '%I/O error%'
       OR json_extract(msg, '$.MESSAGE') LIKE '%degraded%')
ORDER BY "timestamp" DESC
LIMIT 200;

Who logged into the web interface:

SELECT
    "timestamp",
    json_extract(msg, '$._HOSTNAME') AS node,
    json_extract(msg, '$.MESSAGE')   AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'pvedaemon'
  AND json_extract(msg, '$.MESSAGE') LIKE '%authentication failure%'
ORDER BY "timestamp" DESC
LIMIT 100;

Point that last one at an internet-facing node and you will find out how much of the internet is trying root@pam.

Step 6: Alert on failed backups

A query you have to remember to run is a query you will not run. Turn this one into an alert rule, set to alert when there are results:

SELECT 1
FROM logs
WHERE timestamp > CAST(strftime('%s', 'now', '-10 minutes') AS INTEGER)
  AND json_valid(msg)
  AND (json_extract(msg, '$.MESSAGE') LIKE 'ERROR: Backup of VM % failed%'
       OR json_extract(msg, '$.MESSAGE') LIKE 'ERROR: Backup job failed%')
LIMIT 1;

The time window matters. Without it, one failed backup last month keeps the rule firing until retention deletes the line. timestamp is the time Central Logging received the line, in Unix seconds.

Central Logging evaluates rules every five minutes and notifies through Slack, Telegram, Pushover or email. Set a max frequency on the rule. A job that backs up forty guests to a full disk logs forty failures.

Bonus: catch the backup that never starts

The alert above fires when a backup fails. It says nothing when a backup never runs, and a backup job that silently stopped scheduling is worse than one that fails loudly.

Cron monitoring covers that gap. vzdump calls a hook script at each phase of a backup job, so have it ping Central Logging:

#!/bin/bash
# /usr/local/bin/cl-backup-hook.sh
PHASE="$1"
PING="https://logs.example.com/p/YOUR-API-KEY/proxmox-backup"
FAILED=/run/cl-backup-failed

case "$PHASE" in
  job-start)
    rm -f "$FAILED"
    curl -fsS "$PING?state=start" >/dev/null || true ;;
  backup-abort)
    touch "$FAILED" ;;
  job-end)
    if [ -e "$FAILED" ]; then
      curl -fsS "$PING?state=fail&message=one+or+more+guests+failed" >/dev/null || true
    else
      curl -fsS "$PING?state=complete" >/dev/null || true
    fi ;;
  job-abort)
    curl -fsS "$PING?state=fail&message=vzdump+job+aborted" >/dev/null || true ;;
esac

exit 0

job-end runs even when some guests failed to back up. job-abort only runs when the whole job dies. So the script marks a failure at backup-abort, which runs once per failed guest, and reports it at job-end.

Make it executable and register it in /etc/vzdump.conf:

script: /usr/local/bin/cl-backup-hook.sh

Note the || true and the exit 0. A hook that exits non-zero at job-start aborts the whole job. At backup-start it fails that guest’s backup. Do not let a hiccup reaching your log server destroy the thing you were trying to protect.

Central Logging learns how often the job completes from the complete pings. It alerts you when the next one is late, and the moment a fail ping arrives. It needs two completed runs before it can tell that one is late.

Summary

Install the agent on each node for the journal. Add an offset script if you want the full task output. Treat guests as separate machines. Then let SQL and alert rules do the watching.

Central Logging is a single binary you install on a server you already own. It stores logs in SQLite, needs no JVM and no Docker, and costs $187 once. See Deploy to get an instance running, or start with setting up a self-hosted logging server.