0

enter image description hereI am trying to install the ELFK stack on my digital ocean kubernetes cluster but I don't see the logstash index if I do /_cat/indices/.
In the kibana dashboard I can only see metricbeat index pattern and the metrics about my kubernetes cluster. I can also see the logstash deployment on my cluster. Any idea how to configure the logstash to show it up on the Kibana index pattern list?

logstash-deployment.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
  namespace: kube-system
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
  logstash.conf: |
    # all input will come from filebeat, no local logs
    input {
      beats {
        port => 5044
      }
    }
    filter {
      if [message] =~ /^\{.*\}$/ {
        json {
          source => "message"
        }
      }
      if [ClientHost] {
        geoip {
          source => "ClientHost"
        }
      }
    }
    output {
        elasticsearch {
            hosts => [ "elasticsearch-logging:9200" ]
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash-deployment
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:
      containers:
      - name: logstash
        image: docker.elastic.co/logstash/logstash:6.3.0
        ports:
        - containerPort: 5044
        volumeMounts:
          - name: config-volume
            mountPath: /usr/share/logstash/config
          - name: logstash-pipeline-volume
            mountPath: /usr/share/logstash/pipeline
      volumes:
      - name: config-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.yml
              path: logstash.yml
      - name: logstash-pipeline-volume
        configMap:
          name: logstash-configmap
          items:
            - key: logstash.conf
              path: logstash.conf
---
kind: Service
apiVersion: v1
metadata:
  name: logstash-service
  namespace: kube-system
spec:
  selector:
    app: logstash
  ports:
  - protocol: TCP
    port: 5044
    targetPort: 5044
  type: ClusterIP

WOuld appreciate if I get some help on this.

1 Answer 1

0

First, run your logstash with something like:

output {
  stdout { codec => json }
}

When you make sure everything is ok along the way up to the point of output, add the index name to your Logstash:

output {
  elasticsearch {
    hosts => [ "elasticsearch-logging:9200" ]
    index => "MyPreciousData"
  }
}

And don't forget to add some level of security there, user/pass, or API token, whatever you use there:

output {
  elasticsearch {
    hosts => [ "elasticsearch-logging:9200" ]
    index => "MyPreciousData"
    api_key => "iAmThEApIKeY"
    ssl => true
  }
}

And make sure your user have privileges to create indexes.

Once you have your (autocreated) index in, go ahead to Stack Management -> Kibana -> Index Patterns & create one based on that index.

P.S. Next step would be not relying on index autocreation & data type/structure inference, but doing that explicitly

2
  • I don't see any relevance to Istio. Why did you use the Istio tag ? Commented Nov 30, 2021 at 19:56
  • Thank you @Yuri G . WIll try and confirm if it works.
    – test test
    Commented Dec 2, 2021 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.