I was so frustrated from finding no solution to this seemingly common problem that I built a docker image that tails log files and sends them to stdout, to be used as a sidecar container.
Here's what I did:
- I added a volume with
emptyDir{}
to the pod
- I mounted that volume to my main container, with the
mountPath
being the directory to which it writes the logs to
- I added another container to the pod, called "logger", with the image being the log tracker I wrote (
lutraman/logger-sidecar:v2
), and mounted the same volume to /logs
(I programmed the script to read the logs from this directory)
then, all the logs written to that directory, can be accessed by kubectl logs <pod-name> -c logger
Here is an example yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: dummy
labels:
app: dummy
spec:
selector:
matchLabels:
app: dummy
template:
metadata:
labels:
app: dummy
spec:
volumes:
- name: logs
emptyDir: {}
containers:
- name: dummy-app # the app that writes logs to files
image: lutraman/dummy:v2
ports:
- containerPort: 8080
name: http
protocol: TCP
env:
- name: MESSAGE
value: 'hello-test'
- name: LOG_FILE
value: '/var/log/app.log'
volumeMounts:
- name: logs
mountPath: /var/log
- name: logger # the sidecar container tracking logs and sending them to stdout
image: lutraman/logger-sidecar:v2
volumeMounts:
- name: logs
mountPath: /logs
For anyone who is interested, here is how I made the sidecar container:
Dockerfile:
FROM alpine:3.9
RUN apk add bash --no-cache
COPY addTail /addTail
COPY logtrack.sh /logtrack.sh
CMD ["./logtrack.sh"]
addTail:
#!/bin/sh
(exec tail -F logs/$3 | sed "s/^/$3: /" ) &
echo $! >> /tmp/pids
logtrack.sh:
#!/bin/bash
trap cleanup INT
function cleanup() {
while read pid; do kill $pid; echo killed $pid; done < /tmp/pids
}
: > /tmp/pids
for log in $(ls logs); do
./addTail n logs $log
done
inotifyd ./addTail `pwd`/logs:n
/var/log/containers/
. Unless you have rotation enabled, these are persisted even if pods aren't running./var/log/containers/
. So if contents of$HOME/logs/es.log
were also written to stdout you should be able to access them from the worker node it originally ran on.