1

In order to find out if a parameter passed to a function is a "temporary" (only passed into the function) or also referenced outside I use Py_REFCNT. This is done in a C extension package, but for easier reproducibility I decided to provide a Cython implementation based on IPython magic here.

It seems like something changed for functions that accept multiple arguments (it still works as expected for functions that only take one argument) between CPython 3.5 and CPython 3.6:

In [1]: %load_ext cython

In [2]: %%cython
   ...: cdef extern from "Python.h":
   ...:     Py_ssize_t Py_REFCNT(object o)
   ...:
   ...: cpdef func(o, p):
   ...:     return Py_REFCNT(o)

When I run the code on 3.5 it gives me, the expected result:

>>> import numpy as np
>>> func(np.ones(3), np.ones(3))
1

But with 3.6 it gives me 2:

>>> import numpy as np
>>> func(np.ones(3), np.ones(3))
2

In the comments I was asked about the C code so here it is:

static PyObject *
GetRefCount(PyObject *m, PyObject *args) {
    if (PyTuple_CheckExact(args) && PyTuple_Size(args) > 0) {
        Py_ssize_t reference_count = Py_REFCNT(PyTuple_GET_ITEM(args, 0));
        return PyLong_FromSsize_t(reference_count);
    }
    PyErr_SetString(PyExc_TypeError, "wrong input");
    return NULL;
}

And the method definition:

    {"getrefcount",                                     /* ml_name */
     (PyCFunction)GetRefCount,                          /* ml_meth */
     METH_VARARGS,                                      /* ml_flags */
     ""                                                 /* ml_doc */
     },

The results are the same:

>>> import numpy as np
>>> getrefcount(np.ones(3))  # 3.5
1
>>> getrefcount(np.ones(3))  # 3.6
2

I would like to know where (and why) the reference count is incremented in 3.6. I have looked through the CPython source code / the Python issue tracker but I couldn't find an answer.

6
  • In both versions the argument is still being correctly collected when it goes out of scope, which suggests that the generated bytecode must have changed in response to the numeric change. What does diff'ing bytecode tell you?
    – J_H
    Sep 10, 2017 at 18:43
  • @JH Not sure what you mean. Could you elaborate which bytecode you mean?
    – MSeifert
    Sep 10, 2017 at 18:53
  • Say we run a double(n) function in both versions, which just does return 2 * n. You're predicting that n will have a higher ref count in 3.6, cool. I'm suggesting that 3.6 may have generated additional bytecode operations that take another reference, and release it. Could be within the target (double) function, or could be at the call site. So diff'ing decompilations would be instructive.
    – J_H
    Sep 10, 2017 at 19:33
  • Ah, the bytecode for func(np.ones(3), np.ones(3)) is identical in 3.5 and 3.6. But the func itself is a built-in (compiled) function and has no bytecode.
    – MSeifert
    Sep 10, 2017 at 19:35
  • Have you looked at the C generated by Cython? (Can you show us the generated C, both for 3.5 and 3.6?) Sep 10, 2017 at 23:52

1 Answer 1

2

On Python 3.5, the arguments happen to be cleared from the caller's stack by the time your function is executed. On Python 3.6, the arguments happen to still be on the caller's stack as well as in your function's argument tuple.

On Python 3.5, your function call goes through here:

    else {
        PyObject *callargs;
        callargs = load_args(pp_stack, na);
        if (callargs != NULL) {
            READ_TIMESTAMP(*pintr0);
            C_TRACE(x, PyCFunction_Call(func,callargs,NULL));
            READ_TIMESTAMP(*pintr1);
            Py_XDECREF(callargs);
        }
        else {
            x = NULL;
        }
    }

which removes the arguments from the stack to build the argument tuple:

static PyObject *
load_args(PyObject ***pp_stack, int na)
{
    PyObject *args = PyTuple_New(na);
    PyObject *w;

    if (args == NULL)
        return NULL;
    while (--na >= 0) {
        w = EXT_POP(*pp_stack);
        PyTuple_SET_ITEM(args, na, w);
    }
    return args;
}

On 3.6, your function call goes through here:

if (PyCFunction_Check(func)) {
    PyThreadState *tstate = PyThreadState_GET();

    PCALL(PCALL_CFUNCTION);

    stack = (*pp_stack) - nargs - nkwargs;
    C_TRACE(x, _PyCFunction_FastCallKeywords(func, stack, nargs, kwnames));
}

which goes through here

PyObject *
_PyCFunction_FastCallKeywords(PyObject *func, PyObject **stack,
                              Py_ssize_t nargs, PyObject *kwnames)
{
    ...

    result = _PyCFunction_FastCallDict(func, stack, nargs, kwdict);
    Py_XDECREF(kwdict);
    return result;
}

which goes through here:

case METH_VARARGS:
case METH_VARARGS | METH_KEYWORDS:
{
    /* Slow-path: create a temporary tuple */
    ...

    tuple = _PyStack_AsTuple(args, nargs);

    ...
}

which goes through here:

for (i=0; i < nargs; i++) {
    PyObject *item = stack[i];
    Py_INCREF(item);
    PyTuple_SET_ITEM(args, i, item);
}

which leaves the arguments on the stack and builds a tuple with new references to the arguments.

6
  • Thanks for the answer. Do you happen to know why they have to be kept on the stack as well as in the tuple?
    – MSeifert
    Sep 11, 2017 at 1:50
  • @MSeifert: They don't have to be kept on the stack. They just are. Sep 11, 2017 at 2:05
  • Yeah, but as far as I see they wouldn't actually need to be "incremented" when transferred from the "stack array" to the "args tuple", or am I wrong? I mean the references in the tuple were sufficient in 3.5.
    – MSeifert
    Sep 11, 2017 at 2:10
  • @MSeifert: The references aren't transferred in the 3.6 code path; they're copied. Both the stack and the tuple maintain ownership of the references they contain, and both the stack and the tuple are going decref their references. In the 3.5 code path, the references are actually popped off the stack instead of copied. Sep 11, 2017 at 2:26
  • 3
    @MSeifert: Two data structures both own references to the arguments. It is necessary for the refcount to keep track of this accurately. It is probably not necessary for both data structures to have those references. Sep 11, 2017 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.