How to forward rsyslog logs to a remote server over HTTP (omhttp)

Introduction

Classic syslog forwarding moves your problem to a different machine. You still have a text file. You still cannot search it.

rsyslog ships on most Linux distributions and has forwarded logs over UDP port 514 for decades. That path works, and it has two flaws. The far end is a flat file, not a searchable store. And UDP drops messages under load without telling anyone — usually the burst you most wanted to read.

This guide forwards rsyslog to an HTTP ingest endpoint instead. You get structured, queryable logs, and a queue that holds them while the far end is down.

Check what you are running

rsyslogd -v

Which option you use depends on two things: whether your distribution ships omhttp, and which rsyslog version you have.

omhttp first shipped in rsyslog 8.38.0, in 2018. Batching arrived in 8.1903.0. Where it lives depends on the distribution:

There is no package called rsyslog-omhttp on any of them. An earlier version of this guide told you to install it. That was wrong.

Check the version too. Before 8.2510.0, batched omhttp loses a batch when the server answers with a 5xx, unless you also configure retry.ruleset. Below that version, Option B is the simpler safe choice.

Option A: Forward over HTTP with omhttp

Define a JSON template

rsyslog forwards in the traditional syslog format by default, the one from RFC 3164, and then you parse it on the other end. Templates skip that step and emit JSON directly.

Create /etc/rsyslog.d/60-central-logging.conf:

module(load="omhttp")

template(name="cl_json" type="list") {
    constant(value="{")
    constant(value="\"timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"")       property(name="hostname" format="json")
    constant(value="\",\"severity\":\"")   property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"")   property(name="syslogfacility-text")
    constant(value="\",\"tag\":\"")        property(name="programname" format="json")
    constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}

Keep format="json" on the msg and hostname properties. Log messages carry quotes and backslashes all the time. Without escaping you emit malformed lines, ingest drops them, and nothing reports an error.

Add the output action

Append to the same file:

action(
    type="omhttp"
    server="logs.example.com"
    serverport="443"
    usehttps="on"
    restpath="api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
    template="cl_json"

    # Batch rather than one request per message.
    batch="on"
    batch.format="newline"
    batch.maxsize="500"

    # Batches the server rejects outright are written here, not dropped silently.
    errorfile="/var/log/rsyslog-omhttp-errors.log"

    # Spool to disk so a network blip does not lose messages.
    queue.type="LinkedList"
    queue.filename="cl_fwd"
    queue.maxdiskspace="1g"
    queue.saveonshutdown="on"
    queue.size="100000"
    action.resumeRetryCount="-1"
)

Then restart:

sudo rsyslogd -N1        # validate config first
sudo systemctl restart rsyslog

What those queue settings do

Read this block carefully. The defaults are not what you want.

batch="on" with batch.format="newline" packs up to 500 messages into one HTTP request as newline-separated lines, which is the format the ingest endpoint expects. Skip batching and rsyslog opens one HTTPS request per log line. On a busy host, TLS handshakes then swamp everything else.

queue.filename switches on a disk-assisted queue. Without it the queue lives in memory, and an rsyslog restart destroys it. With it, messages spool to rsyslog’s work directory and go out when the endpoint returns. That is /var/lib/rsyslog on Fedora and RHEL. Find yours with grep -ri workdirectory /etc/rsyslog.conf /etc/rsyslog.d/.

action.resumeRetryCount="-1" retries forever. The default is 0, which does not retry at all. Pair this with the disk queue and your setup survives a reboot of the central server.

Retrying covers a server that is down or answering 5xx. It does not cover a 4xx. omhttp treats any 4xx except 429 as permanent and discards the batch. A wrong source token gets exactly that: Central Logging answers 400 Bad Request. Nothing spools and nothing retries. The errorfile line above is how you find out.

queue.maxdiskspace="1g" caps the damage. If the far end stays down long enough to spool a gigabyte, rsyslog discards rather than filling your disk. Size it against your free space and the length of outage you want to survive.

Filtering what gets sent

Forward everything from every host and you drown. Filter before the action:

# Only warnings and above from all facilities.
if $syslogseverity <= 4 then {
    action(type="omhttp" ... )
}

# Or: everything from specific programs.
if $programname == "sshd" or $programname == "sshd-session" or $programname == "sudo" then {
    action(type="omhttp" ... )
}

Severity numbers run 0 (emergency) through 7 (debug), so <= 4 means warning and above.

The sshd rule names two programs on purpose. Since OpenSSH 9.8, login lines come from sshd-session, and a rule on "sshd" alone forwards none of them. See checking SSH logs on Linux.

Option B: omprog

No omhttp, or an rsyslog older than 8.2510.0? Use omprog, which pipes messages to an external program. More moving parts, but it runs on any rsyslog 8.x, including stock Debian and Ubuntu.

Write the shipper:

#!/bin/bash
# /usr/local/bin/rsyslog-to-cl.sh
# omprog feeds this script one log line per line on stdin.
# It batches them and POSTs each batch to the ingest endpoint.
ENDPOINT="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
BATCH_SIZE=200
FLUSH_SECONDS=5
ERRLOG=/var/log/rsyslog-to-cl.log

buf=()

flush() {
    [ ${#buf[@]} -eq 0 ] && return
    delay=1
    # Hold the batch until the server takes it. While we wait, rsyslog
    # queues new lines behind us instead of us throwing them away.
    until printf '%s\n' "${buf[@]}" \
            | curl -sSf -X POST "$ENDPOINT" --data-binary @- -o /dev/null 2>>"$ERRLOG"; do
        sleep "$delay"
        [ "$delay" -lt 60 ] && delay=$((delay * 2))
    done
    buf=()
}

trap flush EXIT

while true; do
    IFS= read -r -t "$FLUSH_SECONDS" line
    rc=$?
    if [ $rc -eq 0 ]; then
        buf+=("$line")
        [ ${#buf[@]} -ge "$BATCH_SIZE" ] && flush
    elif [ $rc -gt 128 ]; then
        flush          # read timed out: send whatever we have
    else
        break          # stdin closed: the EXIT trap sends the rest
    fi
done

The until loop is the important part. A failed POST keeps the batch and tries again, backing off to once a minute. An earlier version of this script ended the curl line with || true, which threw away every batch that failed. Do not copy that from anywhere.

A wrong token makes the loop retry forever. That is deliberate. Lines back up in rsyslog’s disk queue, /var/log/rsyslog-to-cl.log fills with curl: (22) ... error: 400, and nothing is lost while you fix it.

Keep the read timeout. Without it, a quiet host holds lines until it fills a whole batch. On a low-traffic machine that means logs arriving hours late.

sudo chmod +x /usr/local/bin/rsyslog-to-cl.sh

Then in /etc/rsyslog.d/60-central-logging.conf, add a template that ends each line with a newline. The script reads line by line. Without the \n, omprog adds one itself and complains about it in rsyslog’s log every 30 seconds:

template(name="cl_json_nl" type="list") {
    constant(value="{")
    constant(value="\"timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"")       property(name="hostname" format="json")
    constant(value="\",\"severity\":\"")   property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"")   property(name="syslogfacility-text")
    constant(value="\",\"tag\":\"")        property(name="programname" format="json")
    constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}\n")
}

module(load="omprog")

action(
    type="omprog"
    binary="/usr/local/bin/rsyslog-to-cl.sh"
    template="cl_json_nl"
    queue.type="LinkedList"
    queue.filename="cl_prog"
    queue.saveonshutdown="on"
)

Option C: Keep classic syslog forwarding

Already have a fleet forwarding to a central rsyslog box over omfwd? Leave every host alone and convert at the collector:

# On the collector, receive from the fleet.
module(load="imtcp")
input(type="imtcp" port="514")

# Then forward everything upward over HTTP.
action(type="omhttp" server="logs.example.com" ... )

One config change on one machine. That is the least disruptive migration you will find.

The collector needs omhttp, so the version and packaging table above applies to it. Only one machine has to meet it. A Fedora or RHEL collector in front of a Debian fleet works fine. So does Option B on the collector.

Use imtcp rather than imudp for the fleet-to-collector hop. UDP syslog drops messages under load and says nothing, and load spikes during incidents. You lose the exact burst you needed.

Verifying it works

Generate a test message:

logger -t testing "hello from $(hostname)"

It should land in your log source within seconds. If it does not, check rsyslog’s own diagnostics:

sudo journalctl -u rsyslog -n 50
sudo cat /var/log/rsyslog-omhttp-errors.log   # batches the server rejected

Then look in the work directory from earlier. Queue files piling up there mean the endpoint is unreachable or answering 5xx.

An empty queue and no logs arriving usually means the token is wrong. The server answers 400, omhttp drops the batch, and only the errorfile records it.

Querying syslog data

The JSON template above puts every field within reach:

SELECT
    "timestamp",
    json_extract(msg, '$.host')     AS host,
    json_extract(msg, '$.tag')      AS program,
    json_extract(msg, '$.message')  AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.severity') IN ('err', 'crit', 'alert', 'emerg')
ORDER BY "timestamp" DESC
LIMIT 200;

Failed SSH logins, worth watching on anything internet-facing:

SELECT
    json_extract(msg, '$.message') AS message,
    COUNT(*)                       AS attempts
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.tag') IN ('sshd', 'sshd-session')
  AND json_extract(msg, '$.message') LIKE '%Failed password%'
GROUP BY message
ORDER BY attempts DESC
LIMIT 25;

Give that query a threshold and save it as an alert rule. Now you have brute-force detection and you installed nothing new.

Summary

On Fedora and RHEL, use omhttp with batching, a disk-assisted queue and an errorfile. On Debian, Ubuntu, or any rsyslog older than 8.2510.0, use omprog with the retrying script.

The queue settings are the whole game. They decide whether your forwarder keeps the logs from the incident you needed them for, or throws them away. Set them once and stop worrying.

Have an existing syslog fleet? Convert at the collector. Do not touch every host.

Central Logging is a single binary. See Deploy to stand up the receiving end.