0

Recently I have made a plan to develop a Online Judge system (like spoj). And I have write a judge script in python. It forks every time when it receives one submitted code, and then compile the code and use subprocess.Popen to start the program. But when I get program's memory usage with resource.getrusage(resource.RUSAGE_CHILDREN), it returns memory of the forked python process instead of the subprocess.

Below is some of my code:

try:
    programThread = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, preexec_fn = self.limit, bufsize = -1)
except OSError:
    self.status.value = -2
    return -2
self.JudgeID.value = programThread.pid
programOutput = programThread.communicate(_in)
self.status.value = 0
Res = resource.getrusage(resource.RUSAGE_CHILDREN)
self.cpuusage.value = Res.ru_utime + Res.ru_stime
self.memoryusage.value = Res.ru_maxrss * resource.getpagesize()

cpuusage and memoryusage is shared memory created by multiprocessing.Value, the value of memoryusage is 12M. But via top, i've found, that the memory usage of forked python process is 12M while the subprocess is 900K. Are there anything wrong with my code? Sorry for my bad English.

1 Answer 1

3

This is called maxrss because it's the maximum of the RSS usage of the child. The problem is that subprocess uses the standard way to create the subprocess: it first forks, creating a new process id (childpid), and then execv the intended subprocess' executable (which doesn't change the childpid). Then resource returns the maximum of the RSS among the lifetime of this childpid. If this child ran both a forked copy of Python at 12MB and another program at 900K, then the result is 12MB.

I don't really see how to fix this problem...

1
  • 1
    You may need to execute from Python a small wrapper program written in C, which itself executes the real process, waits for it, and finally gets the RUSAGE_CHILDREN. This way, the reported value will be the maximum of the size taken by this small wrapper program and the real program, rather than the maximum of the size taken by Python and the real program.
    – Armin Rigo
    Dec 14, 2012 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.