1

How to calculate elastic search server disc space, how much nodes required for 4TB logs per day.

  • How much disc space required to store in elastic-search index?
  • How to calculate number of nodes?
  • Is the index compressed?
  • How much disc space required for 10G logs in row formate from logstash?
  • Does the index is compressed or not?

Is the following true of false and why, if node is configured to use 5 shards

Is storing 10G logs needs 50G disc space for 5 shards node?

1 Answer 1

3

Many questions demanding long answers. Here in short:

  1. Space: depends. Have you binary data? Or uncompressed text? How many replica do you want - maybe 0? If raw text and 0 replica: space you need is most likely < your original data size.
  2. Maybe one node is enough for your scenario. Or You have 1000 hits/s ? Then you need maybe more nodes. (Nearly linear scalable => adding nodes if need arises).
  3. Yes, default is set to compressed since ES version 0.90 or so
  4. Normally you also want to store the data to disk, not only in memory (although would be possible) => see 1.
  5. see 3.

It's not space=nodes*rawData . It's more like space=compress(rawData)*replicas.

Key to success is to experiment on your own to find out what fits to your needs (these are really varying from scenario to scenario).

See also these answers for better understanding:

Shards and replicas in Elasticsearch

ElasticSearch - Optimal number of Shards per node

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.