21

Is it possible for a .NET 3.5 application to tell the .NET runtime: "hey, I'm going to use n MB memory later on, so please either commit that much now or fail now?"

The context for this is: I have a C# console application that runs a database query that returns a lot of data and and then does some processing on it. The query can take a very long time (hours) and memory usage keeps increasing as results are read. After the query finishes there is an immediate spike in memory due to the processing I need to do. If the machine doesn't have enough RAM the application fails at that point - after wasting hours on the query! This is very frustrating to the user. If there is not enough RAM I'd like the application to fail quickly.

Of course, I could try some hack like allocating a large array that I don't really need and then setting it to null just before I really need the memory, but this is not ideal, because it might actually cause the process to run out of memory when it would otherwise have enough. Ideally, I'd like to use no more memory than needed, but simply fail early on unless a certain amount can be guaranteed for the entire time my application runs. Is this possible?

11
  • Are you sure that your processing is CPU-bound on data in memory? For something to take hours it seems like a lot of IO must also be happening? Commented Jan 22, 2015 at 23:26
  • 2
    This is an interesting problem, to be sure. Perhaps it would be possible for you to cache the query results to secondary storage (disk) or a local database prior to processing. Then, if something goes awry, you could retrieve quickly from local machine. Just a thought.
    – STLDev
    Commented Jan 22, 2015 at 23:32
  • 2
    The basic issue here is that you have no idea how much memory you'll need. So any attempt to "reserve" it up front is doomed. You'll need to make that query code smarter and not consume resources without bound. A 64-bit operating system is otherwise highly indicated, not exactly a problem these days. Commented Jan 22, 2015 at 23:50
  • 1
    I don't know exactly how much memory will be needed, but I can make a reasonable estimate. By running a simple preliminary query I could estimate, for example, that after the DB query completes it might use 600-800 MB. During the in-memory processing it might go up to 1000-1200 MB. This amount of RAM should normally be available, but I can't control what else happens on the machine. Of course, I'm trying to reduce the amount of memory needed. This question is orthogonal to that - a usability improvement to avoid frustrating the user.
    – EM0
    Commented Jan 23, 2015 at 0:01
  • 2
    You do know that memory is virtual, right? Are you sure your program fails because there's not enough virtual memory? Would it not begin paging instead of failing? Commented Jan 23, 2015 at 3:13

1 Answer 1

20

You could try to use the MemoryFailPoint class:

try
{
    using (new System.Runtime.MemoryFailPoint(20)) // 20 megabytes
    {
        ...
    }
}
catch (InsufficientMemoryException)
{
    ...
}
3
  • 1
    Nice finding. At this point, I would also advice the questioner to persist results from the query and dispose any connection/interface to the db. After results can be reloaded from storage to ensure only fetched data is in memory and there are no other internal caches. Also processing code should be checked to ensure there are no leaks.
    – ceztko
    Commented Jan 22, 2015 at 23:56
  • This looks useful - thanks! MSDN says "The MemoryFailPoint makes no guarantees regarding the long-term availability of the memory during the lifetime of the gate", so it's not foolproof, but still promising.
    – EM0
    Commented Jan 23, 2015 at 0:07
  • Having tried this out for a while, I unfortuantely found it to be too unreliable to be of any practical use (for me). Often, the MemoryFailPoint constructor would throw an exception, but the processing would nevertheless succeed. Other times, MemoryFailPoint would not throw, but the processing would later run out of memory anyway.
    – EM0
    Commented May 23, 2016 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.