1

Fluentd in Docker cannot tail from my log file

My input in /var/log/logf/a.log:

test 1st log
test 2nd log
test 3rd log

and my config in /opt/app/conf/fluent.conf:

<source>
   @type tail
   path /var/log/logf/a.log
   tag test
   read_from_head true
   <parse>
      @type none
      message_key test
   </parse>
</source>

<match test>
   @type stdout
</match>

and my Dockerfile id /opt/app/Dockerfile

FROM fluent/fluentd:v1.11-debian-1
USER root

COPY ./conf/fluent.conf /fluentd/etc
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "3.5.2"]
USER fluent

i run my Dockerfile

$ sudo docker build -t log-app .
$ sudo docker run -d --name logging log-app:latest
$ sudo docker logs -f logging

and I got result stuck, I don't know why

2020-10-26 10:24:58 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-10-26 10:24:58 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.2'
2020-10-26 10:24:58 +0000 [info]: gem 'fluentd' version '1.11.4'
2020-10-26 10:24:58 +0000 [warn]: 'pos_file PATH' parameter is not set to a 'tail' source.
2020-10-26 10:24:58 +0000 [warn]: this parameter is highly recommended to save the position to resume tailing.
2020-10-26 10:24:58 +0000 [info]: using configuration file: <ROOT>
 <source>
   @type tail
   path "/var/log/logf/a.log"
   tag "test"
   read_from_head true
   <parse>
      @type "none"
      message_key "test"
      unmatched_lines
   </parse>
 </source>
 <match test>
    @type stdout
 </match>
</ROOT>
2020-10-26 10:24:58 +0000 [info]: starting fluentd-1.11.4 pid=6 ruby="2.6.6"
2020-10-26 10:24:58 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii- 8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2020-10-26 10:24:59 +0000 [info]: adding match pattern="test" type="stdout"
2020-10-26 10:24:59 +0000 [info]: adding source type="tail"
2020-10-26 10:24:59 +0000 [warn]: #0 'pos_file PATH' parameter is not set to a 'tail' source.
2020-10-26 10:24:59 +0000 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2020-10-26 10:24:59 +0000 [info]: #0 starting fluentd worker pid=15 ppid=6 worker=0
2020-10-26 10:24:59 +0000 [info]: #0 fluentd worker is now running worker=0

I think this is a permission problem, but I'm not sure because this Fluentd not throw an error, can you solve this problem guys?


[SOLVED] completely solved by mr karan shah's explanation

i was solved with docker-compose with mounting volume, below:

in file /opt/app/docker-compose.yaml

version: '2'

services:
  fluentd:
    build: .
    container_name: fl-logging
    volumes:
      - "./conf/:/fluentd/etc:ro"
      - "/var/log/logf:/var/log/logf"

and run the docker compose

 $ sudo docker-compose up -d --build
2
  • How do you start your fluentd container? How do you mount the log folders you want to parse? Oct 26, 2020 at 13:49
  • sudo docker build -t log-app && sudo docker run -d --name logging log-app:latest @AndreasJägle i dont know how to using a mount sir, could you give me a example? but thx you respond my problem sir Oct 26, 2020 at 13:53

1 Answer 1

2

The issue is that you have not mounted the local log files into the Fluentd container for it to be accessible.

Use a command like below.

sudo docker run -d --name logging -v PATHTOYOURLOGFILE:/var/log/logf/ log-app:latest

Read more about volumes here.

You can also use a docker-compose file like below

version: '2.2'
services:
 fluentd:
    build: ./opt/app/
    container_name: fl01
    volumes:
      - "/opt/app/conf/:/fluentd/etc/:ro"
      - "PATHTOYOURLOGFILE:/var/log/logf/"
    networks:
      - elastic
    ports:
      - "9880:9880"
networks:
  elastic:
    driver: bridge
2
  • awesome and complete describe, thx sir, now i m really understand about mounting or volumes functions in docker. Oct 26, 2020 at 20:14
  • @Ganang wahyu: glad that I was able to help. Please accept the answer or upvote it 😀
    – karan shah
    Oct 26, 2020 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.