45

I am converting a csh script to a python script. The script calls a memory-intensive executable which requires a very large stack, so the csh script sets the stacksize to unlimited:

limit stacksize unlimited

When I try to reproduce this script in python, I execute them in a very naive manner, using os.system, e.g.:

os.system('some_executable')

But I do not know how to tell the OS to run these executables with unlimited stacksize. Is there a way to specify stacksize for calls within a python script? Is there some low-level system call that I should be using? And is there a module (similar to shutil) which controls this?

1
  • Attempting to set rlimit_stack after Stack Clash remediations may result in failure or related problems. Also see Red Hat Issue 1463241
    – jww
    Jun 21, 2017 at 16:21

4 Answers 4

45

I have good experience with the following code. It doesn't require any special user permissions:

import resource, sys
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
sys.setrecursionlimit(10**6)

It does however not seem to work with pypy.

If resource.setrlimit doesn't work, you can also try using threads:

sys.setrecursionlimit(10**6)
import threading
threading.stack_size(2**26)
threading.Thread(target=main).start()
6
  • 7
    sys.setrecursionlimit(10**6) will make the Python interpreter segfault if you overflow the stack. Jul 8, 2014 at 15:11
  • 7
    Note that, as of this moment, the resource module is not available for Windows OS. Aug 2, 2014 at 18:21
  • @ofer.sheffer Still better than the accepted answer which uses os.system. Sep 22, 2014 at 17:07
  • 1
    @ThomasAhle I am getting the error no module named resource
    – garg10may
    Sep 25, 2016 at 4:14
  • 3
    @ThomasAhle what is the -1? Did you mean resource.RLIM_INFINITY? Dec 28, 2019 at 19:41
21

You can just use the (u)limit command of your shell, if you want:

os.system('ulimit -s unlimited; some_executable')

Or (probably better) use resource.setrlimit:

resource.setrlimit(resource.RLIMIT_STACK, (resource.RLIM_INFINITY, resource.RLIM_INFINITY))
2
  • 1
    The first option doesn't work for me; I believe it is because os.system executes commands in the basic shell, sh, rather than bash or tcsh. But your second option is exactly what I needed, thank you. (I can't tell who suggested setrlimit first so I am awarding this one, since it includes the explicit code. I am grateful for everyone's answers, though). Feb 21, 2011 at 3:26
  • 1
    Yeah, I should have mentioned that - some shells use limit, some use ulimit, some allow both; the explicit invocation is a much better idea. Feb 21, 2011 at 3:36
13

You're looking for the Python setrlimit interface, resource.RLIMIT_STACK.

Note that standard users cannot raise their hard limits, only root (well, a process with the CAP_SYS_RESOURCE capability (see capabilities(7)) processes can raise their limits; so you may need to use the PAM pam_limits(8) limits.conf(5) file to raise the hard limits for the users in question.

4

You can alter the stack size of the current process via threading.stack_size(), but I don't know if that will be correctly inherited by subprocesses. That interface also requires a specific stack size - "unlimited" isn't an option.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.