How to check Apache error logs

The Apache error log is not where the tutorial said

Upstream Apache ships ErrorLog "logs/error_log". That path is relative to ServerRoot, and every distribution sets ServerRoot somewhere different — or replaces the directive outright.

Three real answers on three systems:

Note the filename changes too: error.log on Debian, error_log on Red Hat. A find for one will not turn up the other.

On Red Hat, ServerRoot is /etc/httpd and the package makes /etc/httpd/logs a symlink to /var/log/httpd. Both paths are correct. Only one is where the file lives.

Stop guessing and ask Apache.

Ask Apache where it is logging

apachectl -S

That prints the answer directly:

ServerRoot: "/etc/httpd"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/httpd/error_log"

Main ErrorLog is already resolved against ServerRoot, so it is the real path, not the logs/error_log from the config file.

Above those lines, apachectl -S lists every virtual host with the config file and line number that defined it:

         port 443 namevhost example.com (/etc/httpd/conf.d/example.conf:12)

That tells you which vhost serves a request and which file to edit. Run it before you conclude anything.

Main ErrorLog is the main server’s log. A vhost with its own ErrorLog writes somewhere else, and the dump does not show that. Grep for the directives to see all of them:

# Debian, Ubuntu
grep -r 'ErrorLog\|CustomLog' /etc/apache2/

# RHEL and derivatives
grep -r 'ErrorLog\|CustomLog' /etc/httpd/

On Debian you will see ${APACHE_LOG_DIR}. That is not a shell variable Apache invents — it comes from /etc/apache2/envvars, where the package sets APACHE_LOG_DIR=/var/log/apache2.

And confirm the config you are reading is the config that is loaded:

apachectl configtest

The vhost rule that hides half your errors

Apache’s own config comment says it plainly, and people still miss it.

Define ErrorLog inside a <VirtualHost> and that host’s errors go there and not to the main error log. Same for CustomLog and the access log. Debian’s shipped default vhost does exactly this:

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

Copy that vhost file to make a second site, change the ServerName and forget the log paths, and both sites now write to the same two files. Change the log paths and forget which you changed, and you have been tailing an empty file.

apachectl -S names the file that defines each vhost. Open that file and read its ErrorLog. Do that before you conclude the error log is broken.

Read the line

The default format is assembled in Apache’s own code, not configured anywhere. A line looks like this:

[Sun Sep 20 10:04:22.123456 2026] [proxy_fcgi:error] [pid 2345:tid 6789] [client 192.0.2.4:51000] AH01067: Failed to read FastCGI header, referer: https://example.com/

Two of those are worth knowing about.

[module:level] tells you who is complaining. [ssl:warn] is TLS. [proxy_fcgi:error] is PHP-FPM. [core:error] is Apache itself. When a module you do not recognise is flooding the log, that is the module to turn down.

AHnnnnn is a stable id. Apache builds these with a macro that stamps a fixed five-digit number onto the message. The prose around it changes between versions; the id does not. Search for the id, not the sentence. It is also the right thing to paste into a search engine.

At debug and the trace levels, Apache prefixes the message with the C source file and line that emitted it. Useful once, confusing the first time.

What LogLevel is hiding

LogLevel warn is the default on every distribution and on a stock build.

Apache’s levels, with the numbers it compares internally:

A message is dropped when its number is greater than the configured level. So LogLevel warn keeps warnings and everything more severe, and throws away info, debug and all eight trace levels.

One exception, and it is in the code rather than the docs: a message at exactly notice is never suppressed, whatever LogLevel says. That is why startup and shutdown lines appear in an error log set to crit.

Raise the level for one module instead of the whole server:

LogLevel warn ssl:info proxy_fcgi:debug

That syntax is Apache 2.4 and later. It is the difference between a readable log and 400 MB of TLS handshake detail.

Reload to apply:

sudo apachectl configtest && sudo systemctl reload apache2   # Debian, Ubuntu
sudo apachectl configtest && sudo systemctl reload httpd     # RHEL and derivatives

Turn the module back down when you are done. An info error log on a busy server is large, and trace8 will fill a disk.

The access log is a different file, in a different format on every distribution

While you are here: combined does not mean the same thing everywhere. Three shipped configs, three definitions.

Upstream httpd:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog "logs/access_log" common

Note the active line uses common, not combined. A stock build from source logs no referer and no user agent until you change that.

Debian and Ubuntu:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

%O instead of %b. Different number: %b is response body bytes, %O is everything Apache wrote to the socket including headers. Two servers, same format name, figures that do not add up.

Fedora and RHEL:

CustomLog "logs/access_log" combined_ext

combined_ext is a Red Hat nickname. It appends PID: %P %{tid}P %T — process, thread, and the request duration in seconds. Handy, and it breaks every parser written for the standard format.

Check yours before you write anything that reads the file:

grep -r 'LogFormat\|CustomLog' /etc/apache2/ /etc/httpd/ 2>/dev/null

Two more details in those format strings.

%b prints -, not 0, when no body was sent. That is the Common Log Format convention. %B prints 0. A sum over %b in awk silently skips every 304.

Use %>s, not %s. Bare %s reports the status of the original request; %>s reports the final one. With an ErrorDocument or any internal redirect those differ, and %s tells you about a request the client never saw. Every shipped config uses %>s for this reason.

Three reasons a request is not in the access log

CustomLog ... env=VAR — the line is written only when that environment variable is set. This is how health checks get excluded, and how the request you want gets excluded with them.

CustomLog ... expr=... — the same thing with an expression. If the expression errors, Apache logs a warning to the error log and drops the line.

BufferedLogs On — Apache holds lines in memory and writes them in batches. tail -f looks stalled and is not.

When nginx is in front

If Apache sits behind nginx, HAProxy or a cloud load balancer, %h is the proxy on every single line. The client is in X-Forwarded-For.

Log it directly:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" proxied

Or enable mod_remoteip, set RemoteIPHeader X-Forwarded-For and a RemoteIPTrustedProxy for each proxy you actually run, and %h becomes the client again.

Set the trusted list. The module’s own source comment is blunt about what happens otherwise: with no RemoteIPTrustedProxy or RemoteIPInternalProxy configured, “all proxies will be considered as external trusted proxies”. So RemoteIPHeader on its own means anyone can send an X-Forwarded-For header and write whatever address they like into your access log.

Nginx’s realip module defaults the other way — with no set_real_ip_from it does nothing. If you run both, do not assume the safe default carries over. Reading Nginx access logs has that side.

Two places the error log is not

Startup failures. If Apache dies before it opens the error log, the message goes to stderr and lands in the journal:

journalctl -u apache2 --since "10 minutes ago"   # Debian, Ubuntu
journalctl -u httpd --since "10 minutes ago"     # RHEL and derivatives

Check this whenever systemctl start fails and the error log has nothing new. Viewing one service’s logs with journalctl -u covers reading that.

Containers. The official httpd image rewrites the shipped config at build time. Every ErrorLog becomes /proc/self/fd/2 and every CustomLog becomes /proc/self/fd/1, so there is no log file at all — the lines go to Docker:

docker logs --tail 100 my-apache

Viewing Docker container logs covers the rest.

One server is easy. Four is not.

A customer reports a 500. It hit one of your web servers, and you do not know which, so you open four terminals and grep four files. Last month’s errors are in a rotated, gzipped file on a host you have since rebuilt.

Copy the lines somewhere central and you ask the question once.

Ship the error log

Apache writes to a file. Give the file to the journal and let the CL Agent take it from there. A systemd unit is the least fragile way:

# /etc/systemd/system/apache-errors.service
[Unit]
Description=Ship the Apache error log to the journal
After=apache2.service

[Service]
ExecStart=/usr/bin/tail -F -n 0 /var/log/apache2/error.log
SyslogIdentifier=apache-error
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now apache-errors

tail -F follows the path rather than the file handle, so logrotate does not quietly end the stream. -n 0 starts at the end instead of replaying the whole file.

On Red Hat, change the path to /var/log/httpd/error_log and After= to httpd.service.

Then install the agent and point it at your server:

# /etc/clagent.toml
URL = "https://logs.example.com/api/v1/ingest_logs/YOUR-SOURCE-TOKEN"

Search every web server at once

The agent runs journalctl -o json, so each entry arrives as an object and the Apache line sits in MESSAGE:

SELECT timestamp,
       json_extract(msg, '$._HOSTNAME') AS host,
       json_extract(msg, '$.MESSAGE')   AS line
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'apache-error'
ORDER BY timestamp DESC
LIMIT 100;

json_valid(msg) goes first and stays first. One plain-text line anywhere in the source makes json_extract raise malformed JSON, and SQLite short-circuits AND inside a WHERE clause, so the guard protects the terms after it. It does not protect the SELECT list — the guard has to remove the bad rows, not merely precede the extracts.

Which module is complaining, across the whole fleet:

SELECT substr(json_extract(msg, '$.MESSAGE'),
              instr(json_extract(msg, '$.MESSAGE'), '] [') + 3,
              instr(substr(json_extract(msg, '$.MESSAGE'),
                           instr(json_extract(msg, '$.MESSAGE'), '] [') + 3), ']') - 1) AS tag,
       COUNT(*) AS n
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'apache-error'
GROUP BY tag
ORDER BY n DESC;

tag               n
----------------  -
proxy_fcgi:error  6

Count by message id, which groups errors that have the same cause and different wording:

SELECT substr(json_extract(msg, '$.MESSAGE'),
              instr(json_extract(msg, '$.MESSAGE'), 'AH'), 7) AS code,
       COUNT(*) AS n
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'apache-error'
  AND instr(json_extract(msg, '$.MESSAGE'), 'AH') > 0
GROUP BY code
ORDER BY n DESC;

Keep the instr(...) > 0 guard. Not every line carries an id — third-party modules often skip the macro. When instr finds nothing it returns 0, and substr(line, 0, 7) does not error. It returns the first six characters of the line, so an untagged message shows up in your results as a code called [Sun S. Measured; it is the quietest way to get a wrong answer on this page.

Errors by host, last day:

SELECT json_extract(msg, '$._HOSTNAME') AS host, COUNT(*) AS errors
FROM logs
WHERE json_valid(msg)
  AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'apache-error'
  AND timestamp > CAST(strftime('%s', 'now', '-1 day') AS INTEGER)
GROUP BY host
ORDER BY errors DESC;

One host with all the errors is a sick host. Errors spread evenly is a sick backend.

Get told about the next one

An alert rule fires when its query returns any row. The engine checks whether the result is empty and reads nothing further, so SELECT 1 ... LIMIT 1 is the right shape.

Fire when FastCGI failures pass 10 in ten minutes:

SELECT 1 WHERE (
  SELECT COUNT(*) FROM logs
  WHERE json_valid(msg)
    AND json_extract(msg, '$.SYSLOG_IDENTIFIER') = 'apache-error'
    AND json_extract(msg, '$.MESSAGE') LIKE '%proxy_fcgi:error%'
    AND timestamp > CAST(strftime('%s', 'now', '-10 minutes') AS INTEGER)
) >= 10 LIMIT 1;

Use a scalar subquery for the threshold. SELECT 1 FROM logs WHERE ... HAVING COUNT(*) >= 10 raises HAVING clause on a non-aggregate query, and a rule that errors does not warn you. It never fires.

Rules run every five minutes and deliver within a minute. Set a max frequency on the rule, or one bad afternoon pages you every five minutes for three hours. More in alerting on errors in your logs.

Keep the file from filling the disk

Both distributions ship a logrotate config: /etc/logrotate.d/apache2 or /etc/logrotate.d/httpd. A build from source ships nothing, and the error log grows until the disk does not.

Apache holds the renamed file open after a rotation, so the rotation has to tell it to reopen. The packaged configs do this with a postrotate that reloads Apache. If you write your own, keep that step. Without it Apache keeps writing to the rotated file and the new error.log stays at zero bytes — which is the other way this page’s opening symptom happens.

rotatelogs is the alternative, and it is the one to use when Apache is not under systemd:

ErrorLog "|/usr/bin/rotatelogs -l /var/log/apache2/error.%Y-%m-%d.log 86400"

Apache pipes to it and it handles the rollover itself. No reload, no held file handle.

Where to go next

• Fixing /var/log filling up — finding what is using the space, including the deleted files du cannot see.
• Checking Nginx error logs — the same job on the other web server
• Reading Nginx access logs — the access-log half, including the awk mistake everyone makes
• Sending Apache logs to a central server — the full shipping setup
• Querying logs with SQL — the query language in more depth
• Alerting on errors in your logs — rules that fire when they should
• Viewing one service’s logs with journalctl -u — when Apache will not start at all
• Viewing Docker container logs — when Apache runs in a container

Central Logging is a single binary you install on a server you already own. See Deploy to get an instance running in about five minutes.