Sending Apache logs to a central server

Introduction

Apache writes access_log and error_log to the machine that produced them. That is the whole problem.

You have three web servers behind a load balancer. A customer reports a 502 at 14:06. Which box served them? You do not know, so you SSH into all three and grep each one.

This guide fixes that. It covers making Apache log JSON, getting those lines off the box, and querying them once they land.

Step 1: Log JSON instead of combined

Apache’s default combined format is a space-separated string. Parsing it means writing a regex, and the regex breaks the first time a user agent contains a quote.

Log JSON instead. Add this to /etc/apache2/apache2.conf (Debian, Ubuntu) or /etc/httpd/conf/httpd.conf (RHEL, Fedora):

LogFormat "{\"time\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"remote_addr\":\"%a\",\"host\":\"%V\",\"request_method\":\"%m\",\"request_uri\":\"%U%q\",\"status\":%>s,\"bytes_sent\":%B,\"duration_us\":%D,\"referer\":\"%{Referer}i\",\"user_agent\":\"%{User-Agent}i\"}" json_combined

Then point the access log at it:

CustomLog ${APACHE_LOG_DIR}/access.log json_combined

Test and reload:

sudo apachectl configtest && sudo systemctl reload apache2

One line per request, ready to query:

{"time":"2026-08-07T09:14:02-0700","remote_addr":"203.0.113.9","host":"example.com","request_method":"GET","request_uri":"/pricing","status":200,"bytes_sent":18422,"duration_us":41233,"referer":"https://example.com/","user_agent":"Mozilla/5.0"}

Four of those directives are worth knowing:

• %>s is the final status after internal redirects. Plain %s gives you the first one, which is the wrong number on any site using mod_rewrite.
• %B is bytes sent excluding headers, as a number. %b writes - for a zero-byte response, and - is not valid JSON where a number belongs.
• %D is the request duration in microseconds. %T gives whole seconds, which rounds every fast request to 0.
• %V is the server name that actually handled the request. On a box with virtual hosts, that column is how you tell the sites apart.

Where Apache differs from Nginx

Nginx has escape=json. Apache does not.

Apache escapes control characters and quotes in logged values as \xhh sequences. That is not valid JSON escaping. A request carrying a raw control byte in its user agent produces one line that will not parse.

This is rare and it is not fatal. Guard your queries with json_valid(msg) and the bad line is skipped instead of breaking the result. Every query below does that.

Step 2: Get the logs off the server

Two paths. Pick by whether you run systemd.

Option A: Pipe through syslog and let the agent ship it

On a systemd host this is the shortest route. Apache pipes to logger, logger writes to the journal, and the CL Agent already ships the journal upstream.

CustomLog "|/usr/bin/logger -t apache_access -p local6.info" json_combined
ErrorLog  "|/usr/bin/logger -t apache_error -p local6.err"

Reload, then confirm entries are arriving:

journalctl -t apache_access -n 5 -o json

If the agent is installed, you are finished. Apache logs now flow upstream with the rest of your system logs. The journalctl guide covers how that export works.

Choose the -t tags deliberately. They become SYSLOG_IDENTIFIER in the journal, and that field is how you separate web traffic from everything else at query time.

Piped logging starts a process that lives as long as Apache does. If logger dies, Apache restarts it. If the pipe fills because nothing is draining it, Apache blocks — so keep the journal healthy and set SystemMaxUse= in /etc/systemd/journald.conf on a busy server.

Option B: Tail the file and POST it

Maybe you want Apache to keep writing files. Maybe there is no systemd. Ship the file directly. The ingest endpoint takes newline-separated lines in one request:

curl -X POST https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN \
    --data-binary @/var/log/apache2/access.log

To ship continuously you have to remember where you stopped, or you send the same lines twice. Save the byte offset:

#!/bin/bash
# /usr/local/bin/ship-apache-logs.sh
set -euo pipefail

LOG=/var/log/apache2/access.log
OFFSET_FILE=/var/lib/apache-shipper/offset
ENDPOINT="https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

mkdir -p "$(dirname "$OFFSET_FILE")"
offset=$(cat "$OFFSET_FILE" 2>/dev/null || echo 0)
size=$(stat -c %s "$LOG")

# logrotate truncated or replaced the file; start over.
if [ "$size" -lt "$offset" ]; then
    offset=0
fi

if [ "$size" -eq "$offset" ]; then
    exit 0   # nothing new
fi

tail -c "+$((offset + 1))" "$LOG" \
    | gzip \
    | curl -sf -X POST "$ENDPOINT" --data-binary @-

echo "$size" > "$OFFSET_FILE"

Run it every minute:

* * * * * /usr/local/bin/ship-apache-logs.sh

Three details in that script earn their place.

The rotation check handles logrotate truncating the file underneath you. Without it the shipper goes quiet and nobody notices for a week.

Gzip compresses access logs about 10:1. The server detects the compression from the body itself, so no header is required.

set -euo pipefail means the offset only advances when the upload succeeded. A failed POST retries next minute instead of losing the lines.

If you take this path, monitor the shipper itself with a cron ping. A silent log pipeline looks exactly like a quiet week.

Step 3: Structure the error log too

Apache has an advantage over Nginx here. The error log format is configurable.

ErrorLogFormat "[%{cu}t] [%-m:%l] [pid %P] [client %a] %M"

That gives you a UTC timestamp with microseconds, the module and log level, the pid, and the client address, in a fixed order. It is not JSON — %M is free text — but every field before the message sits in a known place.

Log level matters more than format. The default warn hides the PHP notices and rewrite decisions you will want during an incident. Raising it site-wide is a mistake, though: LogLevel debug on a busy server writes gigabytes an hour. Raise it per module instead:

LogLevel warn rewrite:trace3

Search error logs with full-text search rather than SQL. These three lines are each worth an alert rule:

AH00124: Request exceeded the limit of 10 internal redirects
AH01071: Got error 'PHP message
(70007)The timeout specified has expired

Step 4: Query the access logs

Central Logging stores logs in SQLite and runs SQL over them, so json_extract reaches individual fields.

Slowest endpoints, in milliseconds:

SELECT
    json_extract(msg, '$.request_uri') AS uri,
    COUNT(*)                           AS hits,
    ROUND(AVG(json_extract(msg, '$.duration_us')) / 1000.0, 1) AS avg_ms
FROM logs
WHERE json_valid(msg)
GROUP BY uri
HAVING hits > 10
ORDER BY avg_ms DESC
LIMIT 25;

Every client generating 5xx responses:

SELECT
    "timestamp",
    json_extract(msg, '$.host')        AS site,
    json_extract(msg, '$.remote_addr') AS client,
    json_extract(msg, '$.status')      AS status,
    json_extract(msg, '$.request_uri') AS uri
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.status') >= 500
ORDER BY "timestamp" DESC
LIMIT 200;

Traffic split by virtual host, which is the query you cannot run on a single box:

SELECT
    json_extract(msg, '$.host') AS site,
    COUNT(*)                    AS requests,
    SUM(json_extract(msg, '$.bytes_sent')) AS bytes
FROM logs
WHERE json_valid(msg)
GROUP BY site
ORDER BY requests DESC;

If you shipped through journald

The agent sends journal entries, so your JSON is nested inside the MESSAGE field as a string. Reaching a field takes two hops:

SELECT json_extract(json_extract(msg, '$.MESSAGE'), '$.status') AS status
FROM logs
WHERE json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'apache_access';

Miss that and every query returns nothing, with no error to tell you why.

On sorting by time

The timestamp column is when the server received the line, not when Apache wrote it. They match in normal operation. They do not match after a network outage, when an hour of backlog arrives in one batch and sorts as if it all just happened.

Sort by the request’s own time when the order matters:

ORDER BY json_extract(msg, '$.time') DESC

Step 5: Alert on the ones that matter

A query you have to remember to run is a query you will not run. Turn the useful ones into alert rules.

Start with 5xx. This catches a bad deploy before a customer emails you:

SELECT COUNT(*) AS errors
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.status') >= 500
HAVING errors > 10;

Rules are evaluated every five minutes and notify through Slack, Telegram or Pushover. Set a max frequency on the rule, or a long outage sends you a pager storm instead of one useful alert.

Summary

Log JSON with %>s, %B and %D. Pipe to logger on systemd, or ship the file with an offset script anywhere else. Set ErrorLogFormat so the error log has structure too. Let SQL do the analysis.

You end up with one place to search every web server you run, and no per-GB ingest bill.

Central Logging is a single binary you install on a server you already own. See Deploy to get an instance running in about five minutes.