I have a table with around 100,000 rows at the moment. I want to index the data in this table in a Solr Index.
So the naive method would be to:
- Get all the rows
- For each row: convert to a SolrDocument and add each document to a request
- Once all rows are converted then post the request
Some problems with this approach that I can think of are:
- Loading too much data (the content of the whole table) in to memory
- POSTing a big request
However, some advantages:
- Only one request to the Database
- Only one POST request to Solr
The approach is not scalable, I see that since as the table grows so will the memory requirements and the size of the POST request. I need to perhaps take n
number of rows, process them, then take the next n
?
I'm wondering if any one has any advice about how to best implement this?
(ps. I did search the site but I didn't find any questions that were similar to this.)
Thanks.