Delete a pod and its logs go with it. The pod that crashed is the pod you wanted to read.
kubectl logs --previous buys you one restart. After that the evidence is gone, and a CrashLoopBackOff at 3am has already burned through a dozen restarts by the time you look.
This guide runs a Fluent Bit DaemonSet that reads every container’s log file off each node and posts it to a central server outside the cluster. It works the same on k3s, k0s, kubeadm and managed clusters.
Run your log server on a plain VM, not in the cluster it watches.
This sounds like a detail. It is the whole point. A log server inside the cluster goes down with the cluster, and the moment you most need logs is the moment the cluster is unhealthy. Central Logging is one binary on one machine, so this costs you nothing to get right.
Create a log source in Central Logging and keep the token handy.
Fluent Bit’s Kubernetes filter calls the API server to turn a log file path into a namespace, pod and container name. That needs read access.
apiVersion: v1
kind: Namespace
metadata:
name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources: ["namespaces", "pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: logging
Read-only on namespaces and pods. Nothing else.
Do not put the token in the ConfigMap. ConfigMaps get dumped into issue reports.
kubectl -n logging create secret generic cl-token \
--from-literal=token='YOUR-SOURCE-TOKEN'
The ingest endpoint accepts the token as a bearer header, so it never has to appear in a URL either.
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_logging_*.log
multiline.parser cri
Mem_Buf_Limit 10MB
Skip_Long_Lines On
DB /var/log/flb_kube.db
Refresh_Interval 10
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name http
Match kube.*
Host logs.example.com
Port 443
URI /api/v1/ingest_logs
tls On
Format json_lines
Json_Date_Key time
Json_Date_Format iso8601
Header Authorization Bearer ${CL_TOKEN}
Retry_Limit False
Four lines in there decide whether this works.
Format json_lines emits one JSON object per line. That is exactly what the ingest endpoint expects — it splits the body on newlines and stores one log entry per line. Use plain json instead and Fluent Bit sends an array, which arrives as a single unreadable log entry.
Exclude_Path keeps Fluent Bit from reading its own logs. Without it, one error message produces a log line, which produces an error message. Clusters have died this way.
DB /var/log/flb_kube.db records how far Fluent Bit got in each file. Restart the pod without it and you re-send everything from the beginning.
Retry_Limit False retries forever instead of dropping. Pair it with Mem_Buf_Limit, which caps how much Fluent Bit buffers before it starts dropping anyway. Ten megabytes per node is a reasonable starting point.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
serviceAccountName: fluent-bit
tolerations:
- operator: Exists # run on control-plane nodes too
containers:
- name: fluent-bit
image: fluent/fluent-bit:3.1
env:
- name: CL_TOKEN
valueFrom:
secretKeyRef:
name: cl-token
key: token
resources:
limits:
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: config
mountPath: /fluent-bit/etc/fluent-bit.conf
subPath: fluent-bit.conf
volumes:
- name: varlog
hostPath:
path: /var/log
- name: config
configMap:
name: fluent-bit-config
Apply it and watch:
kubectl apply -f fluent-bit.yaml
kubectl -n logging logs -l app=fluent-bit --tail=20
Mounting /var/log is enough on containerd and on k3s. /var/log/containers holds symlinks into /var/log/pods, and both live under that one mount.
The blanket toleration is deliberate. Control-plane nodes produce the logs you want during an outage, and a DaemonSet without tolerations skips them.
Central Logging stores logs in SQLite, so json_extract reaches into the structure Fluent Bit sent.
Everything from one namespace:
SELECT
"timestamp",
json_extract(msg, '$.kubernetes.pod_name') AS pod,
json_extract(msg, '$.kubernetes.container_name') AS container,
COALESCE(json_extract(msg, '$.log'), msg) AS line
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.kubernetes.namespace_name') = 'production'
ORDER BY "timestamp" DESC
LIMIT 200;
The COALESCE is there for JSON apps. Merge_Log On with Keep_Log Off parses a JSON line into top-level fields and then deletes log. A plain-text line keeps log. So $.log alone comes back empty for every app that logs JSON. The fallback shows the whole record instead.
Which pods are producing errors, which is the first query of any incident:
SELECT
json_extract(msg, '$.kubernetes.namespace_name') AS ns,
json_extract(msg, '$.kubernetes.pod_name') AS pod,
COUNT(*) AS errors
FROM logs
WHERE json_valid(msg)
AND (json_extract(msg, '$.log') LIKE '%error%'
OR json_extract(msg, '$.level') = 'error')
GROUP BY ns, pod
ORDER BY errors DESC
LIMIT 25;
That query checks two places for a reason. Merge_Log On parses a container’s JSON output into top-level fields, so an application logging structured JSON gets a real level field. An application logging plain text keeps everything in log.
Which namespace is generating your volume:
SELECT
json_extract(msg, '$.kubernetes.namespace_name') AS ns,
COUNT(*) AS lines
FROM logs
WHERE json_valid(msg)
GROUP BY ns
ORDER BY lines DESC;
Run that one on day two. It will surprise you, and it tells you exactly what to exclude.
A pod that restarts twice a day for a month is a bug nobody filed. You will not find it in the logs you have shipped so far.
CrashLoopBackOff and OOMKilled are status reasons. Kubernetes sets them on the pod. The container never prints them, so they never reach /var/log/containers. A rule that searches container output for either word matches nothing, forever.
Two other logs do record them. The kubelet logs every back-off to the node’s journal. The kernel logs every out-of-memory kill there too. Ship both.
Add a second input to fluent-bit.conf:
[INPUT]
Name systemd
Tag host.*
Path /var/log/journal
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _TRANSPORT=kernel
Read_From_Tail On
DB /var/log/flb_journal.db
Then change the output’s Match kube.* to Match *, so both inputs reach the server.
The two filters are joined with OR, which is Fluent Bit’s default. You get the kubelet’s lines and the kernel’s, and nothing else from the journal.
Path matters inside a pod. Without it, Fluent Bit only opens journal files written under the pod’s own machine ID. That ID is not the node’s, so it finds nothing. The DaemonSet already mounts /var/log, so /var/log/journal is there.
No /var/log/journal on the node? Its journal is volatile and lives in /run/log/journal. Mount that as well and point Path at it.
kubelet.service is the unit name on kubeadm-built nodes. k3s and some managed distributions may run the kubelet differently. Check with systemctl list-units | grep -i kube on a node, and filter on the unit it shows.
Now the rules. Both are full-text search rules, set to alert when there are results.
Restarts. The kubelet’s line reads Error syncing pod, skipping, and the error inside it names CrashLoopBackOff:
CrashLoopBackOff
The kubelet repeats that line on every retry while the pod is backing off. Set Max Frequency to an hour, or one crash loop messages you every five minutes.
Out-of-memory kills. The kernel writes one line per kill:
"Memory cgroup out of memory"
Alert on that line, not on the oom-kill: summary near it. The kernel rate-limits the summary and never rate-limits the kill line. The kill line names the process and its PID, not the pod. The oom-kill: line, when it appears, carries the pod’s cgroup path in task_memcg=.
Central Logging evaluates rules every five minutes and notifies through Slack or Telegram. A full-text rule only looks at lines that arrived in the last five minutes, so an old crash does not keep it firing.
Kubernetes also records a BackOff event for every restart back-off. Fluent Bit’s kubernetes_events input ships those. It watches the whole cluster, so run it in a single-replica Deployment, not in this DaemonSet. Otherwise every node sends every event. It needs get, list and watch on events, which the default role above does not grant.
The journal route needs no extra RBAC and also catches OOM kills, which produce no event of their own.
Central Logging is built for teams under roughly 10GB a day. A cluster shipping every line from every pod passes that faster than you expect, and most of what it sends is an ingress controller narrating successful requests.
Three filters do most of the work.
Exclude noisy namespaces at the source:
[FILTER]
Name grep
Match kube.*
Exclude $kubernetes['namespace_name'] ^(kube-system|logging)$
Drop health check chatter:
[FILTER]
Name grep
Match kube.*
Exclude log (/healthz|/readyz|kube-probe)
And let pods opt out with an annotation, which K8S-Logging.Exclude On already enabled:
metadata:
annotations:
fluentbit.io/exclude: "true"
Filter at the node. Every line you drop there is one you do not pay for in disk, in query time, or in scrolling.
Say it plainly: if you run a large cluster with a platform team and a real observability budget, Loki or an ELK stack will serve you better. They scale horizontally. Central Logging is one binary on one machine and does not.
This setup is for the k3s cluster in a rack, the three-node cluster behind one product, the homelab. Small enough that a single searchable store is the right answer, and big enough that kubectl logs has stopped being one.
Run the log server outside the cluster. Run Fluent Bit as a DaemonSet with a tail input, the Kubernetes filter and an HTTP output in json_lines format. Keep the token in a Secret. Then filter hard, because a cluster will send you everything if you let it.
Watching pod logs live rather than keeping them? Dozzle does that well and free, and the comparison covers what it does not keep.
Central Logging is a single binary you install on a server you already own. No JVM, no Docker requirement, no per-GB ingest bill — $187 once. See Deploy, or start with setting up a self-hosted logging server.
💌 Get notified on new features and updates