I have an elixir/OTP application that crashes in production due to an out-of-memory issue. The function that cause the crash is called every 6 hours, in a dedicated process. It takes several minutes (~30) to run and looks like this:
def entry_point do
get_jobs_to_scrape()
|> Task.async_stream(&scrape/1)
|> Stream.map(&persist/1)
|> Stream.run()
end
On my local machine I see a constant growth in large binaries memory consumption when the function runs:
Note that when I manually trigger garbage collection on the process that runs the function the memory consumption drops significantly, so it's definitely not a problem with several different processes unable to GC, but only one that doesn't GC properly. In addition, it's important to say that every few minutes the process does manage to GC, but sometimes it's not enough. The production server has only 1GB RAM and it crashes before the GC kicks in.
Trying to solve the issue I came across Erlang in Anger (see pages 66-67). One suggestion is to put all of large binaries manipulations in one-off processes. The return value of the scrape
function is a map that contains the large binaries. Therefore, they are shared between the Task.async_stream
"workers" and the process that runs the function. So, in theory, I could put the persist
together with the scrape
inside the Task.async_stream
. I prefer not to do so, and keep the calls to persist
synchronized through the process.
Another suggestion is to call :erlang.garbage_collect
periodically. It looks like it solves the problem but feels way too hacky. The author also doesn't recommend that. Here's my current solution:
def entry_point do
my_pid = self()
Task.async(fn -> periodically_gc(my_pid) end)
# The rest of the function as before...
end
defp periodically_gc(pid) do
Process.sleep(30_000)
if Process.alive?(pid) do
:erlang.garbage_collect(pid)
periodically_gc(pid)
end
end
And the resulted memory load:
I don't quite understand how the other suggestions in the book fit the problem.
What would you recommend in that case? Keep the hacky solution or there are better options.
persist
function on the data in the table, is that correct? Still, theTask.async_stream
worker will have references to the large binaries, and the main process will have the same references for fetching from ETS inside thepersist
function, and the problem will remain.Task.async_stream
documentation, so if each of yourscrape
calls just build up the binary, stash it in ETS, and return a reference, then you might be in business.Task.async_stream
.