2

How to plan resources (I suspect, elasticsearch instances) according to load:

With load I mean ≈500K events/min, each containing 8-10 fields.

What are the configuration knobs I should turn? I'm new to this stack.

2
  • 1
    How long are you going to keep data around? What kind of query load do you expect you'll have? In the end it'll depend on so many factors that all you can get here is (possibly educated) guesses; you'll simply have to try it out for yourself. May 19, 2015 at 17:57
  • Thanks for comment. The load is forever, retention can be like 2 months. Storage is not an issue here, ability to query is. Querying is for dashboard, 1-2 person should use it at a same time, say I have 20-30 visualizations per dashboard. I just want to know, is it a whole bunch of servers, or it's <10?
    – inteloid
    May 19, 2015 at 18:16

1 Answer 1

5

500,000 events per minute is 8,333 events per second, which should be pretty easy for a small cluster (3-5 machines) to handle.

The problem will come with keeping 720M daily documents open for 60 days (43B documents). If each of the 10 fields is 32 bytes, that's 13.8TB of disk space (nearly 28TB with a single replica).

For comparison, I have 5 nodes at the max (64GB of RAM, 31GB heap), with 1.2B documents consuming 1.2TB of disk space (double with a replica). This cluster could not handle the load with only 32GB of RAM per machine, but it's happy now with 64GB. This is 10 days of data for us.

Roughly, you're expecting to have 40x the number of documents consuming 10x the disk space than my cluster.

I don't have the exact numbers in front of me, but our pilot project for using doc_values is giving us something like a 90% heap savings.

If all of that math holds, and doc_values is that good, you could be OK with a similar cluster as far as actual bytes indexed were concerned. I would solicit additional information on the overhead of having so many individual documents.

We've done some amount of elasticsearch tuning, but there's probably more than could be done as well.

I would advise you to start with a handful of 64GB machines. You can add more as needed. Toss in a couple of (smaller) client nodes as the front-end for index and search requests.

1
  • Thanks. I've got beefy 64Gb RAM machines, will reconsider my retention policy. Suppose I have all that, 10 machines handling 30Tb of data, will elastic cluster be able to query timely, it's roughly 1.5Tb per instance to scan.
    – inteloid
    May 20, 2015 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.