27

Is there any way to force make -j to not over consume my RAM? I work on a dev team, and we have different hardware sets, so -j8 may not be optimal for everyone. However, make -j uses too much RAM for me, and spills over into swap, which can take down my entire system. How can I avoid this?

Ideally, I would want make to watch the system load and stop spawning new threads, wait for some to complete, and continue on.

2
  • make --version reports GNU Make 3.81
    – Drise
    Commented Jul 24, 2012 at 21:33
  • Without verifying, I think this could be solved by having the -j<N> argument set by MAKEFLAGS in people's .bashrc, rather than set in build scripts. Commented Jun 19, 2018 at 15:56

3 Answers 3

10

Is it possible that there is some confusion what make -j does? (at least I had it wrong for a long time...). I assumed that -j without options will adapt to the number of CPUs, but it doesn't - it simply doesn't apply any limit. This means that for big projects it will create a huge number of processes (just run "top" to see...), and possibly use up all the RAM. I stumbled on this when the "make -j" of my project used all of my 16Gb of RAM and failed, while "make -j 8" topped out at 2.5 Gb RAM usage (on 8 cores, load is close to 100% in both cases).

In terms of efficiency, I think using a limit equal to or bigger than the maximum number of CPUs you expect is more efficient that no limit, as the scheduling of thousands of processes has some overhead. Total number of processes created should be constant, but as "-j" creates a lot of them at once, memory allocation might become a problem. Even setting the limit to twice the number of CPUs should still be more conservative that not setting a limit at all.

PS: After some more digging I came up with a neat solution - just read out the number of processors and use that as the -j option:

make -j `nproc`
2
  • This is so underrated! I've got this wrong for years LoL
    – Aetf
    Commented Apr 21, 2021 at 19:34
  • make will still hog memory in this situation (on a Mac Pro). Admittedly, I have found that whatever parallel system (make -j, python3 multiprocessing, GNU-Parallel; haven't tried stress testing joblib with loky) I use on the Mac OS which exceeds some threshold of memory or memory + swap usage will leak over some memory fault (probably) and grab all resources, so it might be a Mac problem. I'll be trying the -l option, but you can always make -n | parallel -k -j `nproc` if -j or -j `nproc` -l 2.5 (or whatever) isn't good enough.
    – Chris
    Commented Jul 22, 2021 at 13:05
6

The somewhat simple solution would be for each workstation to have an environment variable that is suited to what that hardware can handle. Have the makefile read this environment variable and pass it to the -j option. How to get gnu make to read env variables.

Also, if the build process has many steps and takes a long time have make re-read the environment variable so that during the build you can reduce / increase resource usage.

Also, maybe have a service/application running on the workstation that does the monitoring of resource usage and modify the environment variable instead of trying to have make do it...

1
  • 3
    What the hardware can handle depends on what's being compiled and can't be controlled optimally by make -j because the -j option knows nothing about the RAM usage required when building various types of source. For example, -O0 vs -O2 or C vs C++ or GCC vs Intel compilers. Commented Oct 6, 2013 at 17:14
3

It is possible to limit process RAM usage using ulimit. But it may lead to failing process when limit will be exceeded. gcc loves to exceed any limit when linking in single thread. So ulimit solution is not popular.

Another solution is to provide estimate of gcc RAM usage per thread and maintain swap that will be used rarely. You can add load-average to stop spawning of new threads/jobs when load is too high.

I am using the following script /etc/profile.d/makeopts.sh (gentoo):

#!/bin/bash

# We need up to 1000 MB (less than 1GB) per thread.
MAX_THREADS=$(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE) / (1000 ** 3)))
EFFECTIVE_THREADS=$(getconf _NPROCESSORS_ONLN)
THREADS=$((MAX_THREADS < EFFECTIVE_THREADS ? MAX_THREADS : EFFECTIVE_THREADS))

MAX_JOBS=$((MAX_THREADS / THREADS))
JOBS=$((MAX_JOBS < EFFECTIVE_THREADS ? MAX_JOBS : EFFECTIVE_THREADS))

MAX_LOAD=$((EFFECTIVE_THREADS * 9 / 10))

export MAKEOPTS="--jobs=$THREADS --load-average=$MAX_LOAD"
export EMERGE_DEFAULT_OPTS="--jobs=$JOBS --load-average=$MAX_LOAD"

Machine with 4 threads and 16 GB RAM:

MAKEOPTS="--jobs=4 --load-average=3"
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=3"

Machine with 16 threads and 32 GB RAM:

MAKEOPTS="--jobs=16 --load-average=14"
EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=14"

Please be aware that this config was created for CFLAGS="-O2 -pipe -march=native". Please re-estimate it if you want to add/remove light/heavy options.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.