19

What setup works for GNU make parallel jobs (-j) on Windows?

I have tried setting the shell to cmd.exe using MinGW make 3.81, this works in creating the multiple processes but make fails with the "waiting for job" message.

Can this work and what is the best setup? (MinGW / Cygwin / ???) Can someone point me to a working example to test against?

5 Answers 5

5

A tip that might help CMake users

The -jN option does not work when a makefile calls make recursively as cmake-makefiles typically do. But there is a remedy that again works because in the generated makefiles CMake calls via such a syntax.

$(MAKE) -f CMakeFiles\Makefile2 <subproject>

This means you can modify the variable MAKE:

mingw32-make "MAKE=mingw32-make -j3"

Now every time a subproject-make is started, it again gets the "-j3" option. But note that this effectively does not limit the number of parallel compiles to 3 as you might expect. If you have more than 3 projects that don't depend on each other on the same hierarchy then all 3 projects will build parallel and each of them launches 3 compile steps. Resulting in 9 parallel compile steps.

When we take another close look into the top Makefile generated by cmake then we see that the target "all" essentially only starts a sub make.

$(MAKE) -f CMakeFiles\Makefile2 all

So we can remove one layer of subproject parallelism by calling

mingw32-make "MAKE=mingw32-make -j3" -f CMakeFiles\Makefile2 all

But this sacrifices the progress reporting.

I hope this helps nevertheless.

2
  • This works very nicely as a workaround, thanks! I'm pretty sure it's not just useful for cmake users, too :-) - recursive make isn't that unusual. Apr 8, 2013 at 14:14
  • Urk! This sounds like a bad thing to do in a large recursive make system. Every make invocation will be running 3 jobs (for -j3). That could easily run to dozens of simultaneous jobs in total! Note that proper parallel make requires all the make instances to talk to eachother in order to maintain a maximum of 3 jobs (in this case) across all instances. Clever.
    – bobbogo
    Nov 22, 2016 at 13:48
4

As Far as I can understand, GNU Make's parallel jobs depend on the underlying platform to supply (CPU) load information. This is unfortunately done in a non-Windows-compatible way, and only the "make -j" command does anything scalingwise. That command (without max number of jobs) is potentially quite dangerous, as it will eat memory like a rabid starving dog (I tried it on Qt, and it crashed on QtWebkit, the largest project in Qt, 4GB RAM, Win7).

On second thought, I get the feeling make -j3 runs faster than make, but in light of what I can find on the web (forums, manual...) it seems very POSIX/linux specific functionality.

BTW, I'm surprised your question was voted down.

4
  • After almost 10 years I'm still facing the same issue on Qt: if I use make -jn nothing changes, the only possible option is to use make -j without a limit number, but that makes the operating system unusable during the whole compilation process. Are we still without a fix according to you?
    – Brutus
    Aug 29, 2019 at 11:10
  • I do believe newer versions of mingw32-make actually do parallel jobs, but essentially not really well. At least the QtWebEngine stuff secretly uses Ninja to build itself, so that part is definitely faster than before. What make are you running? A lot has changed in the MinGW world since I posted this answer.
    – rubenvb
    Aug 29, 2019 at 11:19
  • Thanks for answering; I'm using Qt Creator to compile a project for an Android target. As NDK I use the release r18b that comes with make.exe v3.81; I read online that make v4 should have a fix for this issue, but I'm not sure where I could download it. If I run make --version I get GNU Make 3.81 - This program built for i686-w64-mingw32. Any hint?
    – Brutus
    Aug 29, 2019 at 14:47
  • I would first really suggest using ninja instead of make if that is an option at all (depends on whatever build system you use for your project). If you're using CMake it's almost trivial (change the generator in Qt Creator kit preferences and make sure ninja is in PATH). I can suggest downloading a recent MinGW-w64 toolchain and try copying its mingw32-make over your make.exe. It doesn't matter what exact version of the toolchain you pick.
    – rubenvb
    Aug 29, 2019 at 14:53
3

I found this Danny Thorpe's blog entry that explains the problems with parallel make build on Windows. Basically what it suggests is to set the following environment variable first:

set SHELL=cmd.exe

With that, I was able to run the make -j style command, but unfortunately not the controlled make -jN style command.

3

I've resolved this problem so I share the solution to everyone.

You can try MinGW-TDM (Home page: http://tdm-gcc.tdragon.net/) or MinGW-Equation (Download page: http://www.equation.com/servlet/equation.cmd?fa=fortran).

Install 32 bit version to D:\MinGW\MinGW32 for example (and 64 bit version to D:\MinGW\MinGW64 if you wish to use 64 bits compilation binaries)

Create a batch file as following then run it at your build directory:

Set MinGW_bin=D:\MinGW\MinGW32\bin
Set MSys_bin=D:\MinGW\msys\bin
Set Path=%MSys_bin%;%MinGW_bin%
mingw32-make SHELL=CMD.exe -j6 -f makefile

You can download MSYS from http://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS. It has many UNIX like utilities that required building many open source libraries, applications.

Where -j6 is the option to run 6 parallel compiling jobs.

TIP: you can set the option -jN where N = your numbers of CPU cores (threads) multiple by 1.5. In my case, my CPU has 4 threads so I set this value to 6.

Note: you can run

make.exe -jN -f makefile

without SHELL=CMD.exe option and your compilation is run parallel but it not compatible in some case (where make.exe is come from MSYS directory).

Regards,

1
  • appears normal MinGW comes with this too, not just TDM, and it works, finally a working parallel make that doesn't just hang using 100% cpu LOL
    – rogerdpack
    Dec 19, 2014 at 14:42
2

I've never had any promblems using make -jn under Cygwin. It works rather well. I regularly use it with Microsoft's cl.exe compiler. It just works out of the box for me. Very UNIX like, which is a Good Thing™. Build scripts port nicely to Linux and the Mac. Thoroughly recommended.

Never liked MinGW make since I got to a place where 5 back-slashes were too few, yet six was too many. Sigh! (To do with MinGW hacks to make backslash separated path names allowable in make AFAIK.)

5
  • Your right. I did get it to work as you describe - I just had to untangle a horrible makefile to find an appropriate point to do -j (make calling make calling make...).
    – Oliver
    Mar 8, 2011 at 20:10
  • if you are using a cygwin/MinGW shell: don't forget to set shell system variable to cmd (otherwise sh or sh.exe will be used by default (on cygwin for example...) and you won't be able to support multi-process execution). just export SHELL=CMD. easy.
    – user257319
    Oct 8, 2015 at 1:24
  • @Karako Noooo, please don't do that. sh just works. It is cross-platform. It has no parallelism issues. (on cygwin anyway—can't vouch for mingw.) cmd is just awful.
    – bobbogo
    Oct 8, 2015 at 13:36
  • Both cygwin and mingw runs parallel make OK, except that some version of mingw has bad multi-thread handling which only runs single-thread make. (Note: I believe the bug is not in the make utility but int the mingw OS itself, for that version.)
    – Robin Hsu
    Nov 22, 2016 at 10:52
  • Make is actually a very non-Unix like tool with all its built-in rules and whatnot.
    – rubenvb
    Feb 9, 2018 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.