I am in a situation of getting Very Huge Amount of Data from Mysql to my springboot Application.
So is there any preferable way to achieve this scenario?
I am in a situation of getting Very Huge Amount of Data from Mysql to my springboot Application.
So is there any preferable way to achieve this scenario?
The big question is: What do you want to do with those 10million records? JPA is mainly good for loading a limited object graph that you then manipulate and flush it back into the database. This only works for a rather limited data volume well. With large amounts of data the overhead become prohibitive and the benefits often vanish.
Without any further knowledge I'd recommend stepping away from JPA and consider more low level ways to access the data.
Especially Springs JdbcTemplate
often comes in handy.
And there if applicable the methods using a RowCallbackHandler
since it doesn't assume that you convert the ResultSet
to anything else but just process it, e.g. in order write out the data to some text file.
Another thing to look at is Spring Batch when processing large amounts of data.
If loading 1 million rows needs 2-3 sec, but loading 10 million rows needs 24 mins, then I think it's a memory problem. JPA maintain a lot of additional ino about the persistent objects, so as long as the objects are inside a transaction the memory need could be much higher than is actually needed for storing the data itself.
So load the data outside of a transaction in smaller chunks (like 1 million), then manually assembly them into one collection. If you need these records for some kind of calculation, then that's all. If you also need to make changes in them, you need to handle the modifications and saves manually.
Although I still believe we can find a better solution if you share the original problem with us...
Pageable
object , Also you can implement fetch type lazy. If you are using data for export report you should use spring batch. – hrdkisback Nov 6 '19 at 7:22