2

I am starting to play with streams in java 8 and I want to do something that feels should be possible but somehow cant seem to get it down.

I have to make call from my service to external service that can process at most 500 unique id and given time. My service can receive any number of this at a given time.

I am looking for a short and precise way to split the list of incoming id's to size of 500 , make call to external service and merge them before returning me the result (Bonus if we can do this in parallel)

private List<ServiceReply> makeServiceCallByBatch(List<Long> ids,AuthToken authToken){
 //make external call and get reply
}

public List<ServiceReply> makeServiceCall(List<Long> ids,AuthToken authToken){
  private int BATCH_SIZE = 500;
  if(ids.size() > BATCH_SIZE){
      //split by batch and process each batch and merge it all
  } else{
      return makeServiceCallByBatch(ids,authToken)
  }
}    
5
  • Did you consider using subList view and performing the operation? Not really sure by the first look, what do you mean by "merge it all"! You might want to have a batchCall at your service implementation to make it cleaner anyway.
    – Naman
    Feb 24, 2020 at 16:47
  • from what little I understand of streams maybe the collect function of it.lets say there was 1400 ids. Process it in 3 batches of 500,500,400 and merge the reply of all 1400 before sending it back.
    – Praveen
    Feb 24, 2020 at 16:50
  • So does that not seem too you like (0,500),(500,1000),(1000,1400)? See a pattern there to split the list? Making a call to your service for those views might just return specific results and then you can addAll to an empty List<ServiceReply> initialized before you start processing.
    – Naman
    Feb 24, 2020 at 16:53
  • that is true. couple things there. 1)This is basic way of doing it. 2)I thought this might be good place to use stream and parallelize external calls . Full disclosure I do not get the full usage of stream and might be over engineering it as it stands
    – Praveen
    Feb 24, 2020 at 16:55
  • Or possibly stackoverflow.com/questions/55105749
    – MikeFHay
    Feb 24, 2020 at 17:19

3 Answers 3

4

List has a sublist(int fromIndex, int toIndex) method that allows you to get a List of just the objects within a specific range in your bigger List. You can use this to pull out Lists from your full List in intervals of 500. So something like-

int index = 0;
int batchSize = 500;
while (index < ids.size()) {
    //used made up method name for doing whatever processing you need to do...
    processBatch(ids.sublist(index, Math.min(index + batchSize, ids.size())), authToken);
    index += batchSize;
}
1
  • Changed the toIndex argument to use Math.min to prevent an IndexOutOfBoundsException
    – cbender
    Feb 24, 2020 at 17:20
0

This is one way to do it:

new ArrayList inputArrayList;

//Write your array to inputArrayList here

while (!inputArrayList.isEmpty()){
    ArrayList newArrayList = new ArrayList();

    for (int i = 0; i < 500 ; i++){
        newArrayList.add(inputArray.lastIndexOf());
        inputArray.remove(inputArray.lastIndexOf());
    }

    // at this point in the while loop, your array of 500 items (newArrayList) should be full. be sure to process it
    // or move it to another array now, or it will be overwritten in the next iteration of the while loop.
}

This code is meant only to show the process. It was written quickly, so it will need tweaking to fit into your app.

0

I'd go with a Stream API based solution:

List<ServiceReply> allReplies = 
    IntStream.iterate(0,                   // start from 0 
                      i -> i < ids.size(), // process until the last element
                      i -> i + BATCH_SIZE) // iterate by BATCH_SIZE
             // extract the sublist from the initial one
             .mapToObj(i -> ids.subList(i, Math.min(i + BATCH_SIZE, ids.size()))
             // make the actual service call
             .map(ids -> makeServiceCallByBatch(ids, authToken))
             // flatten the results into a single stream
             .flatMap(List::stream)
             // collect all the replies into one single list
             .collect(toList());
0

Not the answer you're looking for? Browse other questions tagged or ask your own question.