4

I am trying to use the redis-lua library within copas. It requires some patching. One problem is that redis-lua defines some iterators as coroutines, but these iterators perform network operations that can yield.

So, coroutine.yield is used for two very different things: for the iterator, and for copas. As network calls are nested within iterators, network yields are intercepted by the coroutine.wrap of the iterator, instead of being intercepted by copas.

The example below shows the problem :

local function iterator ()
  for i = 1, 2 do
    if i == 2 then coroutine.yield () end -- network yield
    coroutine.yield () -- iterator yield
  end
end
local citerator = coroutine.wrap (iterator)

local function loop () -- use of the iterator within a copas thread
  while citerator () do end
end
local cloop = coroutine.create (loop)

while coroutine.resume (cloop) do end -- same as copas loop, executes the cloop thread

Is there a "standard" solution to this problem, still allowing to use coroutines for iterators?

I was able to make a small example work by "tagging" the yields (see below), but it is incompatible with existing code. I can leave the copas code unmodified, but have to update iterators in redis-lua.

local function wrap (f, my_tag)
  -- same as coroutine.wrap, but uses my_tag to yield again
  local co = coroutine.create (f)
  return function ()
    local t = table.pack (coroutine.resume (co))
    local code = t [1]
    local tag  = t [2]
    table.remove (t, 1)
    table.remove (t, 1)
    if tag == nil then
      return
    elseif my_tag == tag then
      return table.unpack (t)
    else
      coroutine.yield (tag, table.unpack (t))
    end
  end
end

local Iterator = {} -- tag for iterator yields
local Network  = {} -- tag for network yields

local function iterator ()
  for i = 1, 2 do
    if i == 2 then coroutine.yield (Network, i) end
    coroutine.yield (Iterator, i)
  end
end

local citerator = wrap (iterator, Iterator)

local function loop ()
  while citerator () do end
end

local cloop = wrap (loop, Network)

while cloop () do end

Is there a better solution?

2 Answers 2

5

Lua coroutine always yield to the last thread they were resumed from. The Copas socket functions expect to yield back to the Copas event loop, but instead they get stuck with the coroutine used to implement the redis-lua iterators. Unfortunately there isn't much you can do to fix that except changing the code of the redis-lua iterators. The reason why no one has done so yet is that until Lua 5.2 (LuaJIT can do it as well) it wasn't even possible to yield from an iterator function (the iterator yields in redis-lua work fine because they never leave the iterator function, but you cannot yield beyond the for loop like the Copas socket functions would try to do).

Your idea about using a tag value for distinguishing the iterator yields from the rest is good. You just have to make sure that you pass all yields not intended for the iterator function to the coroutine one level up, including any arguments/return values of coroutine.yield and coroutine.resume (the later is implicit when calling the coroutine.wraped function).

More specifically, if you have code like this in redis-lua:

-- ...
return coroutine.wrap( function()
  -- ...
  while true do
    -- ...
    coroutine.yield( some_values )
  end
end )

You change it to:

-- ...
local co_func = coroutine.wrap( function()
  -- ...
  while true do
    -- ...
    coroutine.yield( ITERATOR_TAG, some_values ) -- mark all iterator yields
  end
  return ITERATOR_TAG -- returns are also intended for the iterator
end )
return function()
  return pass_yields( co_func, co_func() ) -- initial resume of the iterator
end

The ITERATOR_TAG and the pass_yields function go somewhere near the top of redis.lua:

local ITERATOR_TAG = {} -- unique value to mark yields/returns

local function pass_yields( co_func, ... )
  if ... == ITERATOR_TAG then -- yield (or return) intended for iterator?
    return select( 2, ... ) -- strip the ITERATOR_TAG from results and return
  else
    -- pass other yields/resumes back and forth until we hit another iterator
    -- yield (or return); using tail recursion here instead of a loop makes
    -- handling vararg lists easier.
    return pass_yields( co_func, co_func( coroutine.yield( ... ) ) ) 
  end
end

AFAIK, the redis-lua developers are planning to tag another release by the end of the year, so they probably will be grateful for pull requests.

2
  • Using your contribution, i wrote a small module to (almost) transparently replace lua's coroutine one. It passes the lua test suite. The only change is to require the coroutine module as: local coroutine = require "coroutine.make" (). Nov 26, 2014 at 16:15
  • @AlbanLinard Nice idea. I chose a different approach for automating this sort of thing: I wrote a function that automatically transforms iteration functions using callbacks (like the old table.foreach from Lua 5.0) into for-loop iterators using a coroutine and tagged yields.
    – siffiejoe
    Nov 26, 2014 at 17:49
0

In your first example, you are using wrap(loop). I suppose this is copas' wrap, because there is no reference to copas in this code...

However, you are supposed to copas.wrap() a socket, but your loop is a function!

See copas documentation for a good introduction.

1
  • I have edited the first code listing. The wrap is in fact a coroutine.create, and the call to cloop is now coroutine.resume. Nov 26, 2014 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.