Classic syslog forwarding moves your problem to a different machine. You still have a text file. You still cannot search it.
rsyslog ships on most Linux distributions and has forwarded logs over UDP port 514 for decades. That path works, and it has two flaws. The far end is a flat file, not a searchable store. And UDP drops messages under load without telling anyone — usually the burst you most wanted to read.
This guide forwards rsyslog to an HTTP ingest endpoint instead. You get structured, queryable logs and delivery you can trust.
rsyslogd -v
You want 8.x. The omhttp output module used below ships with rsyslog 8.1901.0 and later. On Debian and Ubuntu it lives in a separate package:
sudo apt install rsyslog-omhttp
On RHEL and derivatives:
sudo dnf install rsyslog-omhttp
No omhttp for your distribution? Skip to the omprog approach, which works on any rsyslog 8.x.
rsyslog forwards in RFC 5424 syslog format by default, and then you parse it on the other end. Templates skip that step and emit JSON directly.
Create /etc/rsyslog.d/60-central-logging.conf:
module(load="omhttp")
template(name="cl_json" type="list") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname" format="json")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="programname" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
Keep format="json" on the msg and hostname properties. Log messages carry quotes and backslashes all the time. Without escaping you emit malformed lines, ingest drops them, and nothing reports an error.
Append to the same file:
action(
type="omhttp"
server="logs.example.com"
serverport="443"
usehttps="on"
restpath="api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
template="cl_json"
# Batch rather than one request per message.
batch="on"
batch.format="newline"
batch.maxsize="500"
# Spool to disk so a network blip does not lose messages.
queue.type="LinkedList"
queue.filename="cl_fwd"
queue.maxdiskspace="1g"
queue.saveonshutdown="on"
queue.size="100000"
action.resumeRetryCount="-1"
)
Then restart:
sudo rsyslogd -N1 # validate config first
sudo systemctl restart rsyslog
Read this block carefully. The defaults are not what you want.
batch="on" with batch.format="newline" packs up to 500 messages into one HTTP request as newline-separated lines, which is the format the ingest endpoint expects. Skip batching and rsyslog opens one HTTPS request per log line. On a busy host, TLS handshakes then swamp everything else.
queue.filename switches on a disk-assisted queue. Without it the queue lives in memory, and an rsyslog restart destroys it. With it, messages spool to /var/spool/rsyslog/ and go out when the endpoint returns.
action.resumeRetryCount="-1" retries forever. The default gives up after a few attempts and throws the messages away. Pair this with the disk queue and your setup survives a reboot of the central server.
queue.maxdiskspace="1g" caps the damage. If the far end stays down long enough to spool a gigabyte, rsyslog discards rather than filling your disk. Size it against your free space and the length of outage you want to survive.
Forward everything from every host and you drown. Filter before the action:
# Only warnings and above from all facilities.
if $syslogseverity <= 4 then {
action(type="omhttp" ... )
}
# Or: everything from specific programs.
if $programname == "sshd" or $programname == "sudo" then {
action(type="omhttp" ... )
}
Severity numbers run 0 (emergency) through 7 (debug), so <= 4 means warning and above.
No omhttp? Use omprog, which pipes messages to an external program. More moving parts, but it runs on any rsyslog 8.x.
Write the shipper:
#!/bin/bash
# /usr/local/bin/rsyslog-to-cl.sh
# rsyslog's omprog feeds log lines to this script on stdin, one per line.
# We batch them and POST to the ingest endpoint.
ENDPOINT="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"
BATCH_SIZE=200
FLUSH_SECONDS=5
buf=()
flush() {
[ ${#buf[@]} -eq 0 ] && return
printf '%s\n' "${buf[@]}" \
| curl -sf -X POST "$ENDPOINT" --data-binary @- >/dev/null || true
buf=()
}
trap flush EXIT
while true; do
IFS= read -r -t "$FLUSH_SECONDS" line
rc=$?
if [ $rc -eq 0 ]; then
buf+=("$line")
[ ${#buf[@]} -ge "$BATCH_SIZE" ] && flush
elif [ $rc -gt 128 ]; then
flush # read timed out — send whatever we have
else
break # stdin closed; the EXIT trap flushes the remainder
fi
done
Keep the read timeout. Without it, a quiet host holds lines until it fills a whole batch. On a low-traffic machine that means logs arriving hours late.
sudo chmod +x /usr/local/bin/rsyslog-to-cl.sh
Then in /etc/rsyslog.d/60-central-logging.conf:
module(load="omprog")
action(
type="omprog"
binary="/usr/local/bin/rsyslog-to-cl.sh"
template="cl_json"
queue.type="LinkedList"
queue.filename="cl_prog"
queue.saveonshutdown="on"
)
Already have a fleet forwarding to a central rsyslog box over omfwd? Leave every host alone and convert at the collector:
# On the collector, receive from the fleet.
module(load="imtcp")
input(type="imtcp" port="514")
# Then forward everything upward over HTTP.
action(type="omhttp" server="logs.example.com" ... )
One config change on one machine. That is the least disruptive migration you will find.
Use imtcp rather than imudp for the fleet-to-collector hop. UDP syslog drops messages under load and says nothing, and load spikes during incidents. You lose the exact burst you needed.
Generate a test message:
logger -t testing "hello from $(hostname)"
It should land in your log source within seconds. If it does not, check rsyslog’s own diagnostics:
sudo journalctl -u rsyslog -n 50
ls -la /var/spool/rsyslog/ # messages piling up here means delivery is failing
A growing spool directory means one of two things. The endpoint is unreachable, or the token is wrong.
The JSON template above puts every field within reach:
SELECT
"timestamp",
json_extract(msg, '$.host') AS host,
json_extract(msg, '$.tag') AS program,
json_extract(msg, '$.message') AS message
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.severity') IN ('err', 'crit', 'alert', 'emerg')
ORDER BY "timestamp" DESC
LIMIT 200;
Failed SSH logins, worth watching on anything internet-facing:
SELECT
json_extract(msg, '$.message') AS message,
COUNT(*) AS attempts
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.tag') = 'sshd'
AND json_extract(msg, '$.message') LIKE '%Failed password%'
GROUP BY message
ORDER BY attempts DESC
LIMIT 25;
Give that query a threshold and save it as an alert rule. Now you have brute-force detection and you installed nothing new.
Use omhttp with batching and a disk-assisted queue.
The queue settings are the whole game. They decide whether your forwarder keeps the logs from the incident you needed them for, or throws them away. Set them once and stop worrying.
Have an existing syslog fleet? Convert at the collector. Do not touch every host.
Central Logging is a single binary. See Deploy to stand up the receiving end.
💌 Get notified on new features and updates