37

I am running an 8 cores, 32g RAM elasticsearch node with 5 shards, 400 million (small) documents.

Everything works great until I run an agg search, then shards start failing with:

java.lang.OutOfMemoryError: Java heap space

I have changed heap size with: export ES_HEAP_SIZE=16g (also ES_MAX_MEM and ES_MIN_MEM to same)

also changed the yml file for elasticsearch:

bootstrap.mlockall: true

and even (recommended by install documents):

sudo sysctl -w vm.max_map_count=262144

Restart service and still no no impact, still java.lang.OutOfMemoryError: Java heap space

Any other suggestions? other than don't run agg queries?

query is:

https://localhost:9200/my_index_name/_search?search_type=count
{
  "aggs": {
    "distinct_hostname": {
      "cardinality": {
        "field": "hostname"
      }
    }
  }
}
5
  • What version are you running? What aggregation query are you running? Is it possible to limit amount of documents to put in the aggregate result with a filter? Commented Dec 8, 2014 at 14:43
  • Version: 1.1.1, Build: f1585f0, JVM: 1.7.0_72
    – B.P
    Commented Dec 8, 2014 at 15:04
  • The aggregation query is on a field that exists in each document. The query is an attempt to count the unique instances of the contents of this field. Limiting what it is run on would skew the results. I don't anticipate running this query often and wouldn't mind if it ran slow. But when a shard fails, I get inconsistent results.
    – B.P
    Commented Dec 8, 2014 at 15:05
  • Have you looked at this? elasticsearch.org/guide/en/elasticsearch/guide/current/…
    – Felipe
    Commented Dec 8, 2014 at 16:37
  • Take a look here elastic.co/guide/en/elasticsearch/guide/current/… Commented Mar 1, 2021 at 8:29

3 Answers 3

27

I think I have discovered the error. I was using 'service' to run elasticsearch and therefore my environment variables got stripped. I had to update the /etc/default/elasticsearch file with the correcct env variables (specifically the ES_HEAP_SIZE=16g).

So far it's running well and app is not erroring.

18

The correct way to update Java heap size for Elasticsearch 5 is not EXPORT _JAVA_OPTIONS or EXPORT ES_HEAP_SIZE or using command line parameters. As far as I can tell, all of these are overridden by a configuration file in your Elasticsearch install directory, config/jvm.options. To change these settings you need to edit these lines in that file:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g
2
  • 4
    Just a note for Windows users: jvm.options is used to determine Java options when installing elasticsearch service, but after that any change in jvm.options will have no effect until you recreate the service.
    – qbik
    Commented Jan 11, 2017 at 7:34
  • On Elasticsearch 2.3.3 I had to set both ES_HEAP_SIZE and ES_JAVA_OPTS to get any value but the default. This meant I didn't have to use config/jvm.options
    – eebbesen
    Commented Feb 13, 2020 at 20:34
0

Although this is an old question, here is how you fix this in newer versions of ELK (Elasticsearch, Kibana, Logstash) (v 5.0+):

Configure your jvm.options file to allocate more memory on the heap.
The options you will need to configure are the -Xms and -Xmx options, which by default are set to 1 Gigabyte of RAM (-Xms1G and -Xmx1G). The recommended size for this is no more than half of your system RAM. For example, on a PC with 32GB RAM, you would set set the following: -Xms16G and -Xmx16G.

This file can be found at /etc/elasticsearch/jvm.options on most ELK installs.

The docs have much more detailed info: https://www.elastic.co/guide/en/elasticsearch/reference/master/advanced-configuration.html#set-jvm-heap-size

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.