2

I am sending 1000 at a time to Elasticsearch Bulk API

I have a scenario where i want to update a field inside a document which is equivalent to the multiplication of two fields in the same document. Here is how i am building my query via PHP client

{ "update" : { "_id" : "0", "_type" : "type1", "_index" : "index1"} }
{ "script" : { "inline": "ctx._source.x=ctx._source.y*ctx._source.z"} }

I am using PHP client for this which sends 1000 at one go.

For total of 185000 records its taking 30 mins. Is there any way that i can optimize it ?

6
  • Have you tried update_by_query? Jan 3, 2017 at 16:16
  • I cannot use update_by_query in this scenario because i have set of ids on which i have to perform this update query roughly like above 100k. and each item has a unique value for multiplication. Where as update by query works on a set of results and cannot access ctx.source.field for every document individualy.
    – Raheel
    Jan 3, 2017 at 16:37
  • 1
    Update by query starting from 5.0 supports script. As for Ids problem you can split you list in batches and send several update_by_query requests. Anyway reindexing of 100K records will take minutes. If you can't afford that you have to find other approach. Jan 3, 2017 at 16:42
  • Yes i know, but still lets say i have a factor x which is unique for every document. while update_by_query you specify a script and it will apply on all of the documents. There is no way i can define a unique script for every document.
    – Raheel
    Jan 3, 2017 at 16:46
  • 1
    Unique script for each record is exactly one of roots of yours problem. Another one in size of update. Jan 3, 2017 at 16:54

1 Answer 1

2

If yours script is different from document to document only by factor/parameter then you can create appropriate stored script and then use it in partial update with unique parameter value for each document. This way elastic won't bother with script recompile for each document and byte payload of bulk request will be significantly smaller.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.