5

I'm having an issue with SWIG deleting temporary C++ objects too soon.

Example output from a Python test script:

--------------------------------------------------------------------------------
Works as expected:
    b0 =  Buffer(0, 0, 0, )
    b1 =  Buffer(1, 1, 1, )
    b0 =  Buffer(0, 0, 0, 1, 1, 1, )
    y  =  Buffer(0, 0, 0, 1, 1, 1, )
    b1 =  Buffer(1, 1, 1, )
    repr(b0) =  Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6)
    repr(y)  =  Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6)
Funny business:
    Deleting Buffer(id = 2)
    Deleting Buffer(id = 3)
    repr(b2) =  Buffer(id = 2, vector at 0x020bf790, data at 0x00, size = 4257068)
    Deleting Buffer(id = 4)
    repr(b3) =  Buffer(id = 4, vector at 0x02037040, data at 0x0204a4e0, size = 6)
    Deleting Buffer(id = 0)
    Deleting Buffer(id = 1)

The Deleting Buffer(id = X) is being generated from inside Buffer::~Buffer() C++ code, so we can see here that in the Funny business section, the C++ Buffer objects are getting deleted too early! The Python objects b2 and b3 should be holding references to the C++ Buffer objects with id=2 and id=4.

All my code is attached to my blog post about this issue. However, I'll summarize the code here:

Buffer.hpp:

#include <vector>
#include <string>

struct Buffer
{
    Buffer();
    Buffer(const Buffer & copy);
    ~Buffer();

    Buffer & operator=(const Buffer & rhs);

    Buffer & operator<<(const Buffer & rhs);
    Buffer & operator<<(double rhs);

    std::string __str__() const;
    std::string __repr__() const;

    private:

    std::vector<double> _data;
    int                 _id;
};

swig_test.i:

%module swig_test

%include "std_string.i"

%{
    #include "Buffer.hpp"
    #include <iostream>
%}

%ignore Buffer::operator=;

%include "Buffer.hpp"

go_test.py:

from swig_test import Buffer


def zeros(n):
    '''
    Returns a Buffer filled with 'n' zeros.
    '''

    b = Buffer()

    for i in xrange(n):
        b << 0.0

    return b


def ones(n):
    '''
    Returns a Buffer filled with 'n' ones.
    '''

    b = Buffer()

    for i in xrange(n):
        b << 1.0

    return b


def main():

    #--------------------------------------------------------------------------
    # This sections works as expected

    print "-" * 80
    print "Works as expected:"

    b0 = zeros(3)

    print "    b0 = ", b0

    b1 = ones(3)

    print "    b1 = ", b1

    y = b0 << b1

    print "    b0 = ", b0
    print "    y  = ", y
    print "    b1 = ", b1

    print "    repr(b0) = ", repr(b0)
    print "    repr(y)  = ", repr(y)

    #--------------------------------------------------------------------------
    # Funny things are happening here!

    print "Funny business:"

    b2 = zeros(3) << ones(3)

    print "    repr(b2) = ", repr(b2)

    b3 = zeros(3) << 4.0

    print "    repr(b3) = ", repr(b3)


if __name__ == "__main__":
    main()

I've tried a few things with SWIG as outlined on my blog post but my SWIG-foo skills have turned up short.

Save me SO community, you're my only hope!

Update 1

I suspect I have multiple PyObjects holding Buffer * to the same C++ Buffer object, so when the temporary PyObject is garbage collected, it deletes the C++ Buffer * along with it.

So, I think I need a Py_INCREF somewhere, but where?

Update 2

Trying to return by value as suggested by jarod42 breaks the concatenation paradigm, for example:

b = Buffer()
b << 1 << 2 << 3
print b

only produces:

Buffer(1, )

So that's not what I want.

The %newobject directive can be used to free a newly created object (preventing a memory leak) that a function or method creates. In this case, Buffer::operator<< isn't creating a new object.

1
  • If I could upvote on the basis of that Star Wars quote, I would Dec 31, 2014 at 23:58

2 Answers 2

3

After some more searching I came across this thread which eventually lead to a working solution.

Using a typemap(out) in combination with Py_INCREF did the trick.

swig_test.i:

%module swig_test

%include "std_string.i"

%{
    #include "Buffer.hpp"
    #include <iostream>
%}

%ignore Buffer::operator=;

%typemap(out) Buffer & operator<<
{
    if(result) { /* suppress unused warning */ }
    Py_INCREF($self);
    $result = $self;
}

%include "Buffer.hpp"

Now I get the behavior I want (which matches the pure Python implementation that works) and there are no memory leaks.

1

From http://www.swig.org/Doc1.3/Python.html#Python_nn22
and http://www.swig.org/Doc1.3/Python.html#Python_nn30

With non const reference and pointer, python doesn't reserve memory or take ownership. As it is he case for Buffer& operator<<(const Buffer & rhs); (which return *this).

when you do

b2 = zeros(3) << ones(3)

python has no ownership of b2. The temporary (modified) zeros(3) can be garbage collected once the affectation is done, but b2 points to this temporary and so to invalid memory.

A possible workaround would be to return by value (or const reference). maybe the %newobject directive may help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.