I'm following the algo-class.org course, and one of its programming assignment provide a file with format as below:
1 2
1 5
2 535
...
There are over 5 million such lines, I want to read in the file and convert it to a vector of integer vector like this: [[1 2][1 5][2 535]...].
(defn to-int-vector [s]
(vec (map #(Integer/parseInt %) (re-seq #"\w+" s))))
(def ints (with-open [rdr (clojure.java.io/reader "<file>")]
(doall (map to-int-vector (line-seq rdr)))))
So i believe in this way, I'm not holding the entire file in memory, and only generating a large integer vector. But i get OutOfMemoryError from this. I did try to generate a vector of the same size and same format by running rand-int, and that works fine.
Looks like the memory issue is caused by the temp objects generated? What is the ideal way in clojure to handle a case like this?
Update:
yes, i realize that I am hold the entire integer vector. I have raised the heap size and it now works. I'm interested that a vector and 5 million elements(10 million intergers) can take up so much memory -- I have to allocate 3g for the jvm. Is there any other way that will take the memory down?