4

I have a function which basically returns the entire data from the table.How can i implement batch fetching so that the data will be returned in batches of 60,000 rows at a time.

Will the following logic applicable in this scenario?

http://javainnovations.blogspot.com/2008/07/batch-insertion-in-hibernate.html

5 Answers 5

5

To select data in a batch, you can apply data pagination by setting the initial position & the number of results to be fetched for a query.

for(int i=0; i < MAX_SIZE; i = i + BATCH_SIZE){

    List<Object> resultList = entityManager.createQuery(SQL_QUERY).setFirstResult(i).setMaxResults(BATCH_SIZE).getResultList();

    //-- Batch Computation
} 

I have provided sample code, can modify it accordingly.

2
  • what if the data changes in between the individual selects?
    – premek.v
    Sep 17, 2020 at 16:40
  • @premek.v Versioning should take care of it for any update on stale data; update can happen after executing select also; else go for locking records, but it's not in original question. Sep 23, 2020 at 4:29
2

To select the data with hibernate, in hibernate.properties, set the fetch size using the parameter:

hibernate.jdbc.fetch_size= SOME_VALUE

or in the query:

.setFetchSize(SOME_VALUE)

if you are updating data use: the batch_size

hibernate.jdbc.batch_size= SOME_VALUE
0

In hibernate.properties, set the batch size using the parameter

hibernate.jdbc.batch_size= 'your_value'
0

@nayan-wadekar

Updated form of Batch Processing. The previous comment by him will lead to the following Exception mentioned below because while you are fetching the data it won't be able to fetch the correct object type at run time even after explicit typecasting.

java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to Object

Inorder to fetch the Result List of any object type, use this:

for(int i=0; i < MAX_SIZE; i = i + BATCH_SIZE){

    List<AnyObject> resultList = entityManager.createQuery(SQL_QUERY, AnyObject.class).setFirstResult(i).setMaxResults(BATCH_SIZE).getResultList();

    //-- Batch Computation
} 
0

I had the same issue and wanted to fetch the records in batches but without firing separate queries. I came across Vlad's post here. It uses Java 8 Stream for that. Putting the slightly modified original code below.

Stream<Post> postStream = sessionFactory
        .getCurrentSession()
        .createNativeQuery(
            "SELECT p " +
                "FROM post p " +
                "ORDER BY p.created_on DESC", Post.class)
        .setHint( QueryHints.HINT_FETCH_SIZE, 50 )
        .getResultStream();

stream.forEach(x -> System.out.println(x));

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.