Sending Docker container logs to a central server

Introduction

docker rm deletes your logs. That is the whole problem in one sentence.

Docker’s default driver writes each container’s stdout and stderr to a JSON file under /var/lib/docker/containers/, and docker logs reads it back. One container on one host? Fine. After that it stops helping. The logs die with the container. You cannot search across services. Remove the container and the history goes with it.

Fixing this mostly means picking the right logging driver. You do not need a sidecar, a DaemonSet, or a fleet of collectors.

Understanding Docker logging drivers

A container’s logging driver decides where its stdout and stderr go. Four matter here:

Here is the trick. Get logs into the journal or into syslog and you are on a road other people have already paved. Docker-specific tooling stops being necessary.

Option A: The journald driver (recommended on systemd)

Take this path on any modern Linux host. Docker writes to the journal. The CL Agent ships the journal upstream. The journalctl guide describes that mechanism.

Set it as the default for all containers

Edit or create /etc/docker/daemon.json:

{
  "log-driver": "journald",
  "log-opts": {
    "tag": "{{.Name}}"
  }
}

Restart Docker:

sudo systemctl restart docker

Existing containers keep their old driver. Only containers created after the restart pick up the change. Recreate them:

docker compose up -d --force-recreate

The tag option makes this usable. {{.Name}} writes the container name into the journal’s SYSLOG_IDENTIFIER field, which separates your API container’s logs from your worker’s. Skip it and every container lands under one identifier. Then you cannot filter by service at all.

Check that it works:

journalctl CONTAINER_NAME=my-api -n 20 -o json

Or set it per container

To forward only certain containers:

docker run --log-driver=journald --log-opt tag=my-api my-image

In docker-compose.yml:

services:
  api:
    image: my-image
    logging:
      driver: journald
      options:
        tag: "{{.Name}}"

One nice property: docker logs keeps working under the journald driver, because Docker reads the entries back out of the journal. The syslog driver does not do that, and the surprise catches people out.

Option B: The syslog driver

No systemd on the host? Use the syslog driver and point it at your local syslog daemon:

services:
  api:
    image: my-image
    logging:
      driver: syslog
      options:
        syslog-address: "unixgram:///dev/log"
        tag: "{{.Name}}"

Then forward syslog to your central server. The rsyslog guide has that configuration.

One warning: docker logs returns nothing under this driver, because Docker keeps no local copy. Confirm the logs arrive centrally before you depend on it.

Option C: Stream docker logs directly

To forward one container without touching the daemon config, stream its output to the ingest API:

#!/bin/bash
# /usr/local/bin/ship-container-logs.sh
set -euo pipefail

CONTAINER="$1"
ENDPOINT="https://logs.example.com/api/v1/log/YOUR-SOURCE-TOKEN"

docker logs -f --tail 0 --timestamps "$CONTAINER" 2>&1 \
| while IFS= read -r line; do
    curl -sf -X POST "$ENDPOINT" --data-binary "$line" >/dev/null || true
done

Run it as a systemd unit so it restarts when the container does:

# /etc/systemd/system/[email protected]
[Unit]
Description=Ship %i container logs to Central Logging
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/ship-container-logs.sh %i
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now ship-logs@my-api

This sends one HTTP request per log line. Fine for a quiet container, wasteful for a chatty one. Past a few lines per second, switch to Option A. The agent batches. This does not.

Making container logs structured

Container logs are only as good as what your application prints. Log JSON to stdout and everything downstream gets easier, because Central Logging extracts the fields for you.

Most logging libraries have a JSON mode:

# Python, structlog
structlog.configure(processors=[structlog.processors.JSONRenderer()])

// Node, pino — JSON is the default
const logger = require('pino')()
logger.info({ user_id: 42, route: '/checkout' }, 'order placed')

// Go, log/slog
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Info("order placed", "user_id", 42, "route", "/checkout")

The Python logging guide covers application-level logging in more depth.

Querying container logs

Now filter by container and dig into JSON fields:

SELECT
    "timestamp",
    json_extract(msg, '$.SYSLOG_IDENTIFIER') AS container,
    json_extract(msg, '$.MESSAGE')           AS message
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'my-api'
ORDER BY "timestamp" DESC
LIMIT 200;

Error count per container, worth watching right after a deploy:

SELECT
    json_extract(msg, '$.SYSLOG_IDENTIFIER') AS container,
    COUNT(*)                                 AS errors
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.MESSAGE') LIKE '%error%'
GROUP BY container
ORDER BY errors DESC;

Catching containers that die

The line you want is the last one before the container exited. Restart loops hide well, because the container looks “up” every time you check.

Two things help. First, an alert rule on OOM kills, which are otherwise invisible:

oom-kill OR "Out of memory"

Second, host monitoring, so you know the machine itself is healthy.

A note on log rotation

Keeping the default json-file driver anywhere? Cap the file size. Docker does not rotate by default, and an uncapped container log will fill your disk:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

This is one of the most common ways a Docker host runs out of disk. On the journald driver the journal’s own limits apply instead — see SystemMaxUse in /etc/systemd/journald.conf.

Summary

On systemd hosts, use the journald driver with a tag option and let the agent forward everything. That is one config file and no extra processes. Use the syslog driver elsewhere. Save direct streaming for one-off cases.

Central Logging runs as a single binary on a server you already own. See Deploy to get started.