Delete a pod and its logs go with it. The pod that crashed is the pod you wanted to read.
kubectl logs --previous buys you one restart. After that the evidence is gone, and a CrashLoopBackOff at 3am has already burned through a dozen restarts by the time you look.
This guide runs a Fluent Bit DaemonSet that reads every container’s log file off each node and posts it to a central server outside the cluster. It works the same on k3s, k0s, kubeadm and managed clusters.
Run your log server on a plain VM, not in the cluster it watches.
This sounds like a detail. It is the whole point. A log server inside the cluster goes down with the cluster, and the moment you most need logs is the moment the cluster is unhealthy. Central Logging is one binary on one machine, so this costs you nothing to get right.
Create a log source in Central Logging and keep the token handy.
Fluent Bit’s Kubernetes filter calls the API server to turn a log file path into a namespace, pod and container name. That needs read access.
apiVersion: v1
kind: Namespace
metadata:
name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources: ["namespaces", "pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: logging
Read-only on namespaces and pods. Nothing else.
Do not put the token in the ConfigMap. ConfigMaps get dumped into issue reports.
kubectl -n logging create secret generic cl-token \
--from-literal=token='YOUR-SOURCE-TOKEN'
The ingest endpoint accepts the token as a bearer header, so it never has to appear in a URL either.
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/*_logging_*.log
multiline.parser cri
Mem_Buf_Limit 10MB
Skip_Long_Lines On
DB /var/log/flb_kube.db
Refresh_Interval 10
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name http
Match kube.*
Host logs.example.com
Port 443
URI /api/v1/ingest_logs
tls On
Format json_lines
Json_Date_Key time
Json_Date_Format iso8601
Header Authorization Bearer ${CL_TOKEN}
Retry_Limit False
Four lines in there decide whether this works.
Format json_lines emits one JSON object per line. That is exactly what the ingest endpoint expects — it splits the body on newlines and stores one log entry per line. Use plain json instead and Fluent Bit sends an array, which arrives as a single unreadable log entry.
Exclude_Path keeps Fluent Bit from reading its own logs. Without it, one error message produces a log line, which produces an error message. Clusters have died this way.
DB /var/log/flb_kube.db records how far Fluent Bit got in each file. Restart the pod without it and you re-send everything from the beginning.
Retry_Limit False retries forever instead of dropping. Pair it with Mem_Buf_Limit, which caps how much Fluent Bit buffers before it starts dropping anyway. Ten megabytes per node is a reasonable starting point.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
serviceAccountName: fluent-bit
tolerations:
- operator: Exists # run on control-plane nodes too
containers:
- name: fluent-bit
image: fluent/fluent-bit:3.1
env:
- name: CL_TOKEN
valueFrom:
secretKeyRef:
name: cl-token
key: token
resources:
limits:
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: config
mountPath: /fluent-bit/etc/fluent-bit.conf
subPath: fluent-bit.conf
volumes:
- name: varlog
hostPath:
path: /var/log
- name: config
configMap:
name: fluent-bit-config
Apply it and watch:
kubectl apply -f fluent-bit.yaml
kubectl -n logging logs -l app=fluent-bit --tail=20
Mounting /var/log is enough on containerd and on k3s. /var/log/containers holds symlinks into /var/log/pods, and both live under that one mount.
The blanket toleration is deliberate. Control-plane nodes produce the logs you want during an outage, and a DaemonSet without tolerations skips them.
Central Logging stores logs in SQLite, so json_extract reaches into the structure Fluent Bit sent.
Everything from one namespace:
SELECT
"timestamp",
json_extract(msg, '$.kubernetes.pod_name') AS pod,
json_extract(msg, '$.kubernetes.container_name') AS container,
json_extract(msg, '$.log') AS line
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.kubernetes.namespace_name') = 'production'
ORDER BY "timestamp" DESC
LIMIT 200;
Which pods are producing errors, which is the first query of any incident:
SELECT
json_extract(msg, '$.kubernetes.namespace_name') AS ns,
json_extract(msg, '$.kubernetes.pod_name') AS pod,
COUNT(*) AS errors
FROM logs
WHERE json_valid(msg)
AND (json_extract(msg, '$.log') LIKE '%error%'
OR json_extract(msg, '$.level') = 'error')
GROUP BY ns, pod
ORDER BY errors DESC
LIMIT 25;
That query checks two places for a reason. Merge_Log On parses a container’s JSON output into top-level fields, so an application logging structured JSON gets a real level field. An application logging plain text keeps everything in log.
Which namespace is generating your volume:
SELECT
json_extract(msg, '$.kubernetes.namespace_name') AS ns,
COUNT(*) AS lines
FROM logs
WHERE json_valid(msg)
GROUP BY ns
ORDER BY lines DESC;
Run that one on day two. It will surprise you, and it tells you exactly what to exclude.
A pod that restarts twice a day for a month is a bug nobody filed. Catch it with an alert rule using full-text search:
+"CrashLoopBackOff"
Or on out-of-memory kills, which is the failure most often blamed on something else:
SELECT COUNT(*) AS kills
FROM logs
WHERE json_valid(msg)
AND json_extract(msg, '$.log') LIKE '%OOMKilled%'
HAVING kills > 0;
Central Logging evaluates rules every five minutes and notifies through Slack or Telegram. Set a max frequency, or a crash loop will page you every five minutes until you fix it.
Central Logging is built for teams under roughly 10GB a day. A cluster shipping every line from every pod passes that faster than you expect, and most of what it sends is an ingress controller narrating successful requests.
Three filters do most of the work.
Exclude noisy namespaces at the source:
[FILTER]
Name grep
Match kube.*
Exclude $kubernetes['namespace_name'] ^(kube-system|logging)$
Drop health check chatter:
[FILTER]
Name grep
Match kube.*
Exclude log (/healthz|/readyz|kube-probe)
And let pods opt out with an annotation, which K8S-Logging.Exclude On already enabled:
metadata:
annotations:
fluentbit.io/exclude: "true"
Filter at the node. Every line you drop there is one you do not pay for in disk, in query time, or in scrolling.
Say it plainly: if you run a large cluster with a platform team and a real observability budget, Loki or an ELK stack will serve you better. They scale horizontally. Central Logging is one binary on one machine and does not.
This setup is for the k3s cluster in a rack, the three-node cluster behind one product, the homelab. Small enough that a single searchable store is the right answer, and big enough that kubectl logs has stopped being one.
Run the log server outside the cluster. Run Fluent Bit as a DaemonSet with a tail input, the Kubernetes filter and an HTTP output in json_lines format. Keep the token in a Secret. Then filter hard, because a cluster will send you everything if you let it.
Central Logging is a single binary you install on a server you already own. No JVM, no Docker requirement, no per-GB ingest bill — $187 once. See Deploy, or start with setting up a self-hosted logging server.
💌 Get notified on new features and updates