I'm writing a Python module to perform IO on a O_DIRECT context. One of the limitations of O_DIRECT is you must read into a buffer aligned on a 4096 byte boundary for 2.4 and 2.5 kernels, and 2.6 and up will accept any multiple of 512.
The obvious memory allocation candidate for this is posix_memalign(void **memptr, size_t alignment, size_t size)
In my code, I allocate an area like so:
char *buffer = NULL;
int mem_ret = posix_memalign((void**)&buffer, alignment, size);
if (!buffer) {
PyErr_NoMemory();
return NULL;
}
/* I do some stuff here */
free(buffer);
When I compile and import the module with python3.2, this (and the rest of the unshown module) work fine.
When I attempt the same with python2.7 (I'd like to preserve compatibility) it throws the PyErr_NoMemory exception, and mem_ret == ENOMEM
, indicating it was unable to allocate.
Why would the version of Python I compile against affect how posix_memalign operates?
OS: Ubuntu 12.04 LTS
Compiler: Clang + GCC Show same behaviour
UPDATE
I now have a working piece of code, thanks to user694733
However the fact that it works has me even more confused:
#if PY_MAJOR_VERSION >= 3
char *buffer = NULL;
int mem_ret = posix_memalign((void**)&buffer, alignment, count);
#else
void *mem = NULL;
int mem_ret = posix_memalign(&mem, alignment, count);
char *buffer = (char*)mem;
#endif
Can anyone explain why the incorrect first block works under Python3, but not 2.7, and more importantly why the correct second block does not work under Python3?
UPDATE 2
The plot thickens, having settled on the correct form of the code below, I tested on 4 different version of Python.
void *mem = NULL;
int mem_ret = posix_memalign(&mem, alignment, count);
char *buffer = (char*)mem;
if (!buffer) {
PyErr_NoMemory();
return NULL;
}
/* Do stuff with buffer */
free(buffer);
Under Python 2.7: This code operates as expected.
Under Python 3.1: This code operates as expected.
Under Python 3.2: This code generates mem_ret == ENOMEM
and returns NULL for buffer
Under Python 3.3: This code operates as expected.
The Python versions not included in the Ubuntu repositories were installed from the PPA at https://launchpad.net/~fkrull/+archive/deadsnakes
If the version tagged Python binaries are to be believed, the versions I have installed are:
python2.7
python3.1
python3.2mu (--with-pymalloc --with-wide-unicode)
python3.3m (--with-pymalloc)
Could the use of the wide-unicode flag in the default Python3 distribution be causing this error? If so, how is this happening?
For clarity, the ENOMEM
failure to allocate will occur with any variant of malloc()
, even something as simple as malloc(512)
.
char**
tovoid**
is not valid. Pointers might have different sizes. Usevoid *buffer = NULL;
, and later you can assign it tochar*
if you need byte access.size
andalignment
to see if they make sense? Perhaps they're changed as a side-effect of some UB elsewhere in the code.--with-wide-unicode
.In 2.3, pymalloc has had further enhancements and is now enabled by default
This would indicate that PyMalloc should be built into 2.7 and 3.1, yet they lack them
tag, is this an inconsistency or are those versions genuinely built without it? The 2.7 on my machine is the version that ships with Ubuntu LTS.