1

elasticsearch version 6.8.5
ElasticsearchIO 2.15.0
dataflow version 2.15.0

I have 3 nodes in my cluster and in the same machine I have the instance in gcp with 6 cores and 16GB of ram I have set 4gb per node

I have 5 index with 2 shards per index 2 replicas

dataflow send to elasticsearch total of 2K records in 2 minutes to each index in total 10K records

and send me the next error

java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
Document id jnlmbW8BnhRHn7hU2cLS: rejected execution of processing of [10948][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[indexname][0]] containing [3] requests, target allocation id: M40WguEMSTmWedf2c6LuAg, primary term: 1 on EsThreadPoolExecutor[name = C2JZ4zA/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51a06c32[Running, pool size = 6, active threads = 6, queued tasks = 224, completed tasks = 1026]] (es_rejected_execution_exception)

I check the results and records are missing in elasticsearch
for exaple dataflow say process 1850 records and in elastic search there are 1800 any idea?

1 Answer 1

2

It means that you overwhelmed the request queue. Your queue was full and the nodes just started to ignore the new requests. You can try to increse the size of your queue as you can see here If increasing the queue size doesnt work, you have to either increase your processing power (by adding new nodes or improving the existing nodes) or decrease the number of requests in a period of time. At first I recommend you to monitor your nodes and notice what is going on.

I hope it helps you.

5
  • thanks for you reply i have 3 nodes and looks like the docs are few can you recommend me how many nodes i need and the machine size (cpu ram etc) ? Commented Jan 7, 2020 at 20:40
  • 1
    Ive found another alternative for you. Your problem might not be simply the hardware, but the configuration file. Try to increse the size of the threadpool queue in your instances as you can find here elastic.co/guide/en/elasticsearch/reference/current/…
    – rmesteves
    Commented Jan 7, 2020 at 20:51
  • looks good but i didnt find where change this setting is it by api rest ? or file config ? Commented Jan 8, 2020 at 23:13
  • It might help you google.com/url?sa=t&source=web&rct=j&url=https://…
    – rmesteves
    Commented Jan 8, 2020 at 23:17
  • i add more queue to write thread pool and works this param works up to the limits of hardware so we can see if we need more hardware resources if the error reappears this param must be set in all nodes i modified the file elasticsearch.yml in folder /etc/elasticsearch i added this line thread_pool.write.queue_size : 500 Commented Jan 9, 2020 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.