15

I would like a particular set of Python subprocesses to be as low-impact as possible. I'm already using nice to help limit CPU consumption. But ideally I/O would be limited as well. (If skeptical, please humor me and assume there is value in doing this; it doesn't matter how long they take to run, there can be a lot of them, and there is higher-priority stuff (usually) going on on the same machine, etc.)

One possibility appears to be ionice. Are there any existing Python packages for invoking ionice (Google didn't turn up anything)? It wouldn't be difficult to write code to simply run the ionice command; but I'd prefer to avoid writing code that someone else has written/tested; sometimes there are subtle edge cases, etc. And, is there just a better way to limit I/O consumption?

The man page for ionice suggests that the ionice value can be affected by the nice value, but running this Python 2.6 script appears to disprove that, even for child processes where the nice value is inherited:

#!/usr/bin/env python

import os
import multiprocessing

def print_ionice(name):
    print '*** ', name, ' ***'
    os.system("echo -n 'nice: '; nice")
    os.system("echo -n 'ionice: '; ionice -p%d" % os.getpid())

for niced in (None, 19):
    if niced: os.nice(niced)
    print '**** niced to: ', niced, ' ****'
    print_ionice('parent')
    subproc = multiprocessing.Process(target=print_ionice, args=['child'])
    subproc.start()
    subproc.join()

Which has the following output:

$ uname -as
Linux x.fake.org 2.6.27-11-server #1 SMP Thu Jan 29 20:13:12 UTC 2009 x86_64 GNU/Linux
$ ./foo.py
**** niced to:  None  ****
***  parent  ***
nice: 0
ionice: none: prio 4
***  child  ***
nice: 0
ionice: none: prio 4
**** niced to:  19  ****
***  parent  ***
nice: 19
ionice: none: prio 4
***  child  ***
nice: 19
ionice: none: prio 4

3 Answers 3

18

psutil exposes this functionality (python 2.4 -> 3.2):

import psutil, os
p = psutil.Process(os.getpid())
p.ionice(psutil.IOPRIO_CLASS_IDLE)

Also, starting from Python 3.3 this will be available in python stdlib as well: http://bugs.python.org/issue10784

4
  • Cool fix, wish it was available for OSX. Aug 22, 2013 at 21:47
  • yeah... unfortunately OSX just does not expose it natively. Aug 23, 2013 at 10:46
  • Make sure your block device has CFQ scheduler. Otherwise it won't work.
    – Zorg
    Aug 19, 2017 at 1:02
  • 3
    The issue you linked to makes no mention of ionice (or ioprio_get/ioprio_set). That functionality is still missing in the Python standard library, except as implemented by e.g. @tzot 's solution. Only getpriority/setpriority have been implemented.
    – ocket8888
    Apr 5, 2018 at 20:04
5

Hm.

As a start pointer, you should find what syscall number are the ioprio_set and ioprio_get system calls in your kernel. I'd suggest you check in /usr/include/asm/unistd_32.h or /usr/include/asm/unistd_64.h, depending on your kernel arch; if not there, start with the suggestion of the syscall(2) man page, which should be /usr/include/sys/syscall.h and work your way down includes.

Given that, you should use ctypes, à la:

def ioprio_set(which, who, ioprio):
    rc= ctypes.CDLL('libc.so.6').syscall(289, which, who, ioprio)
    # some error checking goes here, and possibly exception throwing

That's it, more or less. Have fun :)

3

Why not have whatever launches the processes do the ionice on them (i.e., run them with ionice) rather than having them ionice themselves? It seems a whole lot cleaner.

2
  • 1
    I effectively want to fork and not exec (e.g., using the 'multiprocessing' module); unsure where ionice would fit in there. The children could call ionice -p<pid> -cblah (on themselves, e.g., using os.system). Apr 1, 2009 at 5:37
  • @JacobGabrielson Since Linux kernel version 2.6.13 child processes will inherit their parent's nice values and receive a default ionice class of 2, which gives it a best-effort priority between 0 and 7 calculated by (nice + 10 ) // 5. So you should be able to control it that way.
    – ocket8888
    Apr 5, 2018 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.