Sending Docker container logs to a central server

Introduction

docker rm deletes your logs. That is the whole problem in one sentence.

Docker’s default driver writes each container’s stdout and stderr to a JSON file under /var/lib/docker/containers/, and docker logs reads it back. One container on one host? Fine. After that it stops helping. The logs die with the container. You cannot search across services. Remove the container and the history goes with it.

Fixing this mostly means picking the right logging driver. You do not need a sidecar, a DaemonSet, or a fleet of collectors.

Understanding Docker logging drivers

A container’s logging driver decides where its stdout and stderr go. Four matter here:

Here is the trick. Get logs into the journal or into syslog and you are on a road other people have already paved. Docker-specific tooling stops being necessary.

Option A: The journald driver (recommended on systemd)

Take this path on any modern Linux host. Docker writes to the journal. The CL Agent ships the journal upstream. The journalctl guide describes that mechanism.

Set it as the default for all containers

Edit or create /etc/docker/daemon.json:

{
  "log-driver": "journald",
  "log-opts": {
    "tag": "{{.Name}}"
  }
}

Restart Docker:

sudo systemctl restart docker

Existing containers keep their old driver. Only containers created after the restart pick up the change. Recreate them:

docker compose up -d --force-recreate

That recreates Compose containers only. A container you started with docker run has to be removed and run again.

The tag option makes this readable. {{.Name}} writes the container name into the journal’s SYSLOG_IDENTIFIER field. Skip it and Docker uses the default tag, {{.ID}}: the first 12 characters of the container ID. That still separates containers. It stops separating services the first time you recreate one, because the new container gets a new ID.

The driver also writes CONTAINER_NAME on every entry, whatever the tag says. Filter on that when you are not sure which tag a container was started with.

Check that it works:

journalctl CONTAINER_NAME=my-api -n 20 -o json

Or set it per container

To forward only certain containers:

docker run --log-driver=journald --log-opt tag=my-api my-image

In docker-compose.yml:

services:
  api:
    image: my-image
    logging:
      driver: journald
      options:
        tag: "{{.Name}}"

One nice property: docker logs keeps working under the journald driver, because Docker reads the entries back out of the journal.

Option B: The syslog driver

No systemd on the host? Use the syslog driver and point it at your local syslog daemon:

services:
  api:
    image: my-image
    logging:
      driver: syslog
      options:
        syslog-address: "unixgram:///dev/log"
        tag: "{{.Name}}"

Then forward syslog to your central server. The guide to forwarding rsyslog to a remote server over HTTP has that configuration.

docker logs still works here, on Docker Engine 20.10 or later. Docker keeps a local cache for any driver that cannot be read back. It is on by default and holds five compressed files of 20 MB each per container. It is a ring buffer, so old lines fall off the end. Treat it as a convenience, not a copy. It can be turned off with the cache-disabled log option.

Option C: Stream docker logs directly

To forward one container without touching the daemon config, stream its output to the ingest API:

#!/bin/bash
# /usr/local/bin/ship-container-logs.sh
set -euo pipefail

CONTAINER="$1"
ENDPOINT="https://logs.example.com/api/v1/log/YOUR-SOURCE-TOKEN"

docker logs -f --tail 0 "$CONTAINER" 2>&1 \
| while IFS= read -r line; do
    curl -sf -X POST "$ENDPOINT" --data-binary "$line" >/dev/null || true
done

Run it as a systemd unit so it restarts when the container does:

# /etc/systemd/system/[email protected]
[Unit]
Description=Ship %i container logs to Central Logging
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/ship-container-logs.sh %i
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now ship-logs@my-api

Leave out --timestamps. It puts a timestamp in front of every line, and a JSON line stops being JSON. Central Logging stores the time it received each line anyway.

This sends one HTTP request per log line. Fine for a quiet container, wasteful for a chatty one. Past a few lines per second, switch to Option A. The agent batches. This does not.

It also has a gap. docker logs -f exits when the container stops, and systemd restarts the script five seconds later. --tail 0 skips anything written in between. The journald driver has no such gap.

Making container logs structured

Container logs are only as good as what your application prints. Log JSON to stdout and everything downstream gets easier, because Central Logging extracts the fields for you in the log view.

Most logging libraries have a JSON mode:

# Python, structlog
structlog.configure(processors=[structlog.processors.JSONRenderer()])

// Node, pino — JSON is the default
const logger = require('pino')()
logger.info({ user_id: 42, route: '/checkout' }, 'order placed')

// Go, log/slog
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Info("order placed", "user_id", 42, "route", "/checkout")

The Python logging guide covers application-level logging in more depth.

Querying container logs

Filter on CONTAINER_NAME. The journald driver sets it on every line, whatever the tag says:

SELECT
    "timestamp",
    json_extract(msg, '$.CONTAINER_NAME') AS container,
    json_extract(msg, '$.MESSAGE')        AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.CONTAINER_NAME') = 'my-api'
ORDER BY "timestamp" DESC
LIMIT 200;

Started it with Compose and no container_name? Then Compose picked the name, something like myproject-api-1. Run docker ps to see it.

Lines written to stderr, per container, over the last hour. Worth watching right after a deploy:

SELECT
    json_extract(msg, '$.CONTAINER_NAME') AS container,
    COUNT(*)                              AS stderr_lines
FROM logs
WHERE timestamp > CAST(strftime('%s', 'now', '-1 hour') AS INTEGER)
  AND json_valid(msg)
  AND json_extract(msg, '$.CONTAINER_ID') IS NOT NULL
  AND json_extract(msg, '$.PRIORITY') = '3'
GROUP BY container
ORDER BY stderr_lines DESC;

Three details make that query right:

• CONTAINER_ID IS NOT NULL keeps it to containers. The agent ships the whole journal, so sshd and the kernel are in the same source.
• The driver writes stderr at priority 3 (err) and stdout at 6 (info). That beats LIKE '%error%', which also counts 0 errors. It is only as good as your app, though. Plenty of apps log everything to stderr.
• timestamp is the time Central Logging received the line, in Unix seconds.

Your app logs JSON? Then MESSAGE holds a JSON string, and SQL has to unwrap it twice:

SELECT json_extract(json_extract(msg, '$.MESSAGE'), '$.route') AS route
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.CONTAINER_NAME') = 'my-api'
  AND json_valid(json_extract(msg, '$.MESSAGE'));

One trap: a container started with a TTY (-t, or tty: true in Compose) ends each line with \r. journald treats that as binary. journalctl -o json then writes MESSAGE as an array of numbers instead of a string, and every LIKE on it misses. Drop the TTY for anything you ship.

Catching containers that die

The line you want is the last one before the container exited. Restart loops hide well, because the container looks “up” every time you check.

Two things help. First, an alert rule on OOM kills, which are otherwise invisible. The kernel logs both kinds with the same words: Out of memory: Killed process for the host, Memory cgroup out of memory: Killed process for a container that hit its own limit. LIKE ignores case, so one pattern catches both:

SELECT 1
FROM logs
WHERE timestamp > CAST(strftime('%s', 'now', '-10 minutes') AS INTEGER)
  AND json_valid(msg)
  AND json_extract(msg, '$.MESSAGE') LIKE '%out of memory: Killed process%'
LIMIT 1;

Set it to alert when there are results. The window matters. Without it, one OOM kill last month keeps the rule firing until retention deletes the line.

Second, host monitoring, so you know the machine itself is healthy.

A note on log rotation

Keeping the default json-file driver anywhere? Cap the file size. Docker does not rotate by default, and an uncapped container log will fill your disk:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

This is one of the most common ways a Docker host runs out of disk. It only applies to containers created after the daemon restarts. Limiting Docker log size covers the containers it skips and the local driver alternative. On the journald driver the journal’s own limits apply instead — see SystemMaxUse in /etc/systemd/journald.conf.

Summary

On systemd hosts, use the journald driver with a tag option and let the agent forward everything. That is one config file and no extra processes. Use the syslog driver elsewhere. Save direct streaming for one-off cases.

Just need to read one container’s output? Viewing Docker container logs with docker logs covers tail, follow, grep and the three reasons it comes back empty. Looking for the file itself? Docker logs location covers where each driver keeps it. If you already watch them in Dozzle, the Dozzle comparison covers what shipping them adds.

Central Logging runs as a single binary on a server you already own. See Deploy to get started.