elasticsearch version 6.8.5
ElasticsearchIO 2.15.0
dataflow version 2.15.0
I have 3 nodes in my cluster and in the same machine I have the instance in gcp with 6 cores and 16GB of ram I have set 4gb per node
I have 5 index with 2 shards per index 2 replicas
dataflow send to elasticsearch total of 2K records in 2 minutes to each index in total 10K records
and send me the next error
java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
Document id jnlmbW8BnhRHn7hU2cLS: rejected execution of processing of [10948][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[indexname][0]] containing [3] requests, target allocation id: M40WguEMSTmWedf2c6LuAg, primary term: 1 on EsThreadPoolExecutor[name = C2JZ4zA/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@51a06c32[Running, pool size = 6, active threads = 6, queued tasks = 224, completed tasks = 1026]] (es_rejected_execution_exception)
I check the results and records are missing in elasticsearch
for exaple dataflow say process 1850 records and in elastic search there are 1800
any idea?