4

I have Python code that has lots of loops that consume data from Python generators. Some also re yield the processed data. This is a bottleneck and I want to speed this part up and was thinking of using Cython.

What is the recommended way to deal with generators and yield. I would like to

  • Convert Python generators into Cython without data copies
  • Make Cython for loops consume data produced by Python generators
  • Yield data like a generator

I would guess this is a common enough use case, what is the recommended ways to do this.

2
  • You'd need to provide more details.. also, please explain why you believe using cython is the way to go. There might be better solutions.
    – shx2
    Jun 1, 2014 at 19:14
  • For anyone led here while searching for the Cython error: "yield not supported here", make sure you are writing a def, not a cpdef: groups.google.com/forum/#!topic/cython-users/DSJ_DlwCHHw
    – 0 _
    Jan 4, 2016 at 16:58

1 Answer 1

8

Some recommendations:

  • Cython supports generators out of the box, so you should try passing your Python code with generators to cython and see what kind of speedup you get.
  • Next step is to add as much static typing information to your loops to speed up the work the generators are doing.
  • Python generators are cool, but if performance is important, they aren't the fastest way to do things. You're much better off converting your bottlenecks to working with contiguous arrays.
  • Check out Cython's typed memoryviews.
  • You can also use Cython with C++ std::vectors and other high-performance container objects.

We'll need more information about your goals and constraints to provide more help here. A stripped down example would be helpful.

5
  • I have pipelines of streams of unknown length. Generators and coroutines leads to much cleaner code in this scenario. I will try out generators with Cython, is this a new addition ?
    – san
    Jun 2, 2014 at 14:41
  • You may have to compromise on the purity of your code to achieve the performance you need. Since C does not natively support generators, Cython has to introduce a fair amount of extra calls to make them work. They will still be faster than pure Python generators, though. Arrays are very fast in C, so you can get much better performance from them.
    – lothario
    Jun 2, 2014 at 15:32
  • 2
    You might consider a hybrid buffered solution: you can store a C array or typed memorview, do batches of computations with them, and then yield the results one by one. This way you can keep your streaming algorithm, but benefit from the performance of arrays and contiguous memory access.
    – lothario
    Jun 2, 2014 at 15:34
  • Yes absolutely, these streams are indeed buffered and yielded exactly as you state. Its really a pity that C chose to leave out the coroutines abstraction. Could you point me to authoritative documentation of yield in Cython. Google hits give me conflicting claims, perhaps because this functionality was added later.
    – san
    Jun 2, 2014 at 18:25
  • Could you point me to authoritative documentation of yield in Cython, thats all I need to close this as an accept, and thanks for your answer.
    – san
    Jun 9, 2014 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.