[Prescript: I know that nothing here is specific to Delayed::Job. But it helps establish the context.]
update
I believe the SQL queries are not being garbage collected. My application generates many large SQL insert/update operations (160K bytes each, about 1 per second) and sends them to PostgreSQL via:
ActiveRecord::Base.connection.execute(my_large_query)
When I perform these db operations, my application slowly grows without bound. When I stub out the db operations (but perform all the other functions in my app) the bloating stops.
So: any ideas on why this is happening, how I can pinpoint it, or how I can make it stop?
original question
I have delayed tasks that slurp data from the web and create records in a PostgreSQL database. They seem to be working okay, but they start at vmemsize=100M and within ten minutes bulk up to vmemsize=500M and just keeps growing. My MacBook Pro with 8G of RAM starts thrashing when the VM runs out.
How can I find where the memory is going?
Before you refer me to other SO posts on the topic:
I've added the following to my #after(job) method:
def after(job)
clss = [Object, String, Array, Hash, ActiveRecord::Base, ActiveRecord::Relation]
clss.each {|cls| object_report(cls, " pre-gc")}
ObjectSpace.each_object(ActiveRecord::Relation).each(&:reset)
GC.start
clss.each {|cls| object_report(cls, "post-gc")}
end
def object_report(cls, msg)
log(sprintf("%s: %9d %s", msg, ObjectSpace.each_object(cls).count, cls))
end
It reports usage on the fundamental classes, explicitly resets ActiveRecord::Relation objects (suggested by this SO post), explicitly does a GC (as suggested by this SO post), and reports on how many Objects / Strings / Arrays / Hashes, etc there are (as suggested by this SO post). For what it's worth, none of those classes are growing significantly. (Are there other classes I should be looking at? But wouldn't that be reflected in the number of Objects anyway?)
I can't use memprof because I'm running Ruby 1.9.
And there are other tools that I'd consider if I were running on Linux, but I'm on OS X.