72

I am trying to memcheck a C python extension I am writing, but I'm having trouble setting up valgrind to work with python. I would really appreciate some advice. Just for context, this is Ubuntu 13.10, python 2.7.5+, and valgrind 3.8.1.

As per recommendation from Readme.valgrind I did the following.

1) Downloaded the python source with

sudo apt-get build-dep python2.7
apt-get source python2.7

2) Applied the code patch, i.e. "Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c".

3) Applied the suppression patch, i.e. "Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc"

4) Compiled python with

./configure --prefix=/home/dejan/workspace/python --without-pymalloc
make -j4 install

Note that I did both 2 and 3, while README.valgrind says to do 2 or 3... more can't hurt.

Now, let's test this on some sample python code in test.py

print "Test"

Let's run valgrind on python with this script

valgrind --tool=memcheck --leak-check=full --suppressions=python2.7-2.7.5/Misc/valgrind-python.supp bin/python test.py

Unexpectedly, there is still loads of reports from valgrind, with the first one being (and many more following)

==27944== HEAP SUMMARY:
==27944==     in use at exit: 857,932 bytes in 5,144 blocks  
==27944==   total heap usage: 22,766 allocs, 17,622 frees, 4,276,934 bytes allocated
==27944== 
==27944== 38 bytes in 1 blocks are possibly lost in loss record 24 of 1,343
==27944==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27944==    by 0x46B8DD: PyString_FromString (stringobject.c:143)
==27944==    by 0x439631: PyFile_FromFile (fileobject.c:157)
==27944==    by 0x4E9B4A: _PySys_Init (sysmodule.c:1383)
==27944==    by 0x4E29E9: Py_InitializeEx (pythonrun.c:222)
==27944==    by 0x4154B4: Py_Main (main.c:546)
==27944==    by 0x577DDE4: (below main) (libc-start.c:260)

Am I doing something wrong? Is there a way to valgrind a python script that doesn't leak and get clean valgrind output?

3
  • Have you seen stackoverflow.com/questions/1519276/…
    – VooDooNOFX
    Nov 21, 2013 at 5:22
  • @VooDooNOFX I've searched stackoverflow and beyond and all answers basically say that I should do what I described I did -- compile python or use a supress. This does get rid of the custom allocation problematic reports such as PyObject_Free (this is the one in the question you linked), but the reports I'm getting are not of that sort. Nov 21, 2013 at 5:28
  • In order to test my c extension, I compiled debug python with instructions provided in the Answer. And I rebuild my extension with debug version python, and it got failure result. I used setuptools extension build the extension, and it works well under system environment. The build flags are very different.
    – Hua
    Dec 14, 2016 at 8:50

2 Answers 2

65

Since python 3.6, there's a PYTHONMALLOC environment variable which is available in release builds, without needing to recompile.

PYTHONMALLOC=malloc python3 foobar.py

This will disable pymalloc and just use the libc malloc directly, making it valgrind-friendly. This is equivalent to --without-pymalloc (and it is just as slow)

If valgrind is too slow, other values can be helpful. PYTHONMALLOC=debug and PYTHONMALLOC=malloc_debug add debug hooks on top of the default and the libc allocators respectively. Their effects, from the docs:

  • Newly allocated memory is filled with the byte 0xCB
  • Freed memory is filled with the byte 0xDB
  • Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
  • Detect writes before the start of a buffer (buffer underflows)
  • Detect writes after the end of a buffer (buffer overflows)
  • Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.

This will catch some uninitialized reads, some use after free, some buffer under/overflows, etc, but won't report leaks and won't touch memory that isn't allocated through python (When using glibc, the MALLOC_PERTURB_ and MALLOC_CHECK_ environment variables might help there)

See also:

0
56

I found the answer here.

Python also needs to be compiled in debug mode, i.e.

./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind

In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.

6
  • 1
    Thanks for doing the research for everyone! Do you know if one should recompile python once done using valgrind? And, more importantly - how to do it?
    – Yair Daon
    Mar 5, 2015 at 15:47
  • What works for me is compiling/installing a valgrind-compatible version of python, then basing a virtualenv on that. That gives me the convenient activate script. Even better with virtualenvwrapper Nov 15, 2016 at 13:48
  • does this make every python command call run with valgrind? I'm noticing a significant slow down installing packages with pip Jan 25, 2017 at 23:34
  • 1
    @JamesRobertAlbert From the README: Note: configuring python --without-pymalloc makes Python run much slower, especially when running under Valgrind. ... It seems to take about 5 times longer to run --without-pymalloc
    – ssnobody
    Feb 10, 2017 at 2:23
  • 1
    @YairDaon you don't have to recompile Python because you create a new interpreter. Related : stackoverflow.com/questions/36697863/….
    – DRz
    May 15, 2018 at 3:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.