9

I am trying to better understand when I should and should not use Iterators. To me, whenever I have a potentially large amount of data to iterate through, I write an Iterator for it. If it also lends itself to the Iterator interface, then it seems like a win.

I was reading a little bit that there is a lot of overhead with using an Iterator.

A good example of where I used an Iterator was to iterate through a bunch of SQL scripts to execute one query at a time, reading it in, then executing it.

Is there another performance trade off I should be aware of? Before I used iterators, I would read the entire String of SQL commands to execute into an ArrayList, and the iterate through that. If the import is rather large (like for geolocation data, then the server tends to get bogged down).

Walter

1

2 Answers 2

9

I think your question is when you should 'stream' input rather than load it all into memory and the process it. It's not really a question of using Iterator or not I think.

"It depends," of course, though in your given example it sounds like streaming the input rather than loading it all into memory is a clear win, so iterate indeed.

The benefit of loading into memory is usually that the code is simpler, and maybe you get some benefit from loading large chunks into memory at once rather than reading bits at a time. The benefit of "streaming" is that you limit your memory requirements, and, gain performance associated with that.

As a very crude rule of thumb, I wouldn't load anything like this into memory unless I were sure it was under 100K or so.

1
  • Thanks for your input - Yeah, that is what I was getting at as well as the validity of the Iterator. I like to be exposed to different approaches to solve the same problem. Both will work, as the files get larger and larger, eventually the Iterator will be more efficient and not consume all the memory in the server.
    – Walter White
    Apr 1, 2010 at 1:08
1

A good example of where I used an Iterator was to iterate through a bunch of SQL scripts to execute one query at a time, reading it in, then executing it.

In this scenario the overhead of an Iterator is likely dwarfed by the time it takes to run the queries.

Before I used iterators, I would read the entire String of SQL commands to execute into an ArrayList, and the iterate through that. If the import is rather large (like for geolocation data, then the server tends to get bogged down).

Any particular reason you need to collect them all into an ArrayList? You could just execute them one by one as you read the statements.

Iterators are particularly suited for streaming cases where the data is loaded/created on the fly/lazily. They do not require the data to be completely in memory upfront.

1
  • I used an ArrayList at that time because I needed to keep the order and it was easier to put the SQL commands into an ArrayList. Even though I've used Iterators, I haven't written one before myself, so I didn't think of writing it that way. Now that I've used them a bit more, it seems like a perfect scenario to use it.
    – Walter White
    Apr 1, 2010 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.