1

I'm working on software using a library with a huge memory usage (e.g. LightGBM).

I'm working on a data-science software which as the property to reduce RAM usage dynamically when data is not asked, and reload it from disk when necessary depending on our needs, kind of an advanced and configurable swap to sum up. Therefore, when I call external code, we except that memory follows kind of the same requirements.

When working on huge dataset, memory usage can go way further available memory, the idea his to limit memory usage to avoid being stuck at 100% memory usage. As soon as I don't want to modify memory management within LightGBM's code because it would mean choose a specific version and re-adapt code each time I want to update. In my software, can I programmatically restrict (and later release) physical RAM usage of my application, to force swapping?

Excepted pseudo-code:

some_function_before();
some_API::please_use_swap(/*threshold=*/16);
some_process_with_heavily_memory_usage();
some_API::end_requirement();
some_function_after();

If there is another approach to resolve this, I'll pick it of course.

Thanks.

10
  • "My software as the property to reduce RAM usage dynamically..." Has it? Or are you trying to achieve that?
    – Yunnosch
    Oct 23, 2020 at 8:06
  • "kind of an advanced and configurable swap to sum up" Advanced how? Configurable how?
    – Yunnosch
    Oct 23, 2020 at 8:07
  • "As soon as I can't handle memory management of LightGBM ..." So at the specifc point of time when that happens..... ?
    – Yunnosch
    Oct 23, 2020 at 8:08
  • 1
    Is this a convoluted way of asking "Can I programmatically restrict (and later release) physical RAM useage of my application, to force swapping?". That is what I understand. But you also want to not change the applications code... I still am lost.
    – Yunnosch
    Oct 23, 2020 at 8:10
  • On an unrelated note, any is a C++ standard library header. So seeing <any> in the question title is a bit confusing (the question body makes no mention of it). So consider editing your title into something clearer. Oct 23, 2020 at 8:12

1 Answer 1

0

There's such an API on Windows: SetProcessWorkingSetSize. You state how much physical RAM you want to use; the rest could be paged out.

As is normal, this is just a hint. Windows may decide that there's plenty of RAM and ignore your hint altogether.

Not the answer you're looking for? Browse other questions tagged or ask your own question.