10

I wanted to have a list of lambdas that act as sort of a cache to some heavy computation and noticed this:

>>> [j() for j in [lambda:i for i in range(10)]]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

Although

>>> list([lambda:i for i in range(10)])
[<function <lambda> at 0xb6f9d1ec>, <function <lambda> at 0xb6f9d22c>, <function <lambda> at 0xb6f9d26c>, <function <lambda> at 0xb6f9d2ac>, <function <lambda> at 0xb6f9d2ec>, <function <lambda> at 0xb6f9d32c>, <function <lambda> at 0xb6f9d36c>, <function <lambda> at 0xb6f9d3ac>, <function <lambda> at 0xb6f9d3ec>, <function <lambda> at 0xb6f9d42c>]

Meaning that the lambdas are unique functions but they somehow all share the same index value.

Is this a bug or a feature? How do I avoid this problem? It's not limited to list comprehensions...

>>> funcs = []
... for i in range(10):
...     funcs.append(lambda:i)
... [j() for j in funcs]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
0

5 Answers 5

18

The lambda returns the value of i at the time you call it. Since you call the lambda after the loop has finished running, the value of i will always be 9.

You can create a local i variable in the lambda to hold the value at the time the lambda was defined:

>>> [j() for j in [lambda i=i:i for i in range(10)]]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Another solution is to create a function that returns the lambda:

def create_lambda(i):
    return lambda:i
>>> [j() for j in [create_lambda(i) for i in range(10)]]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

This works because there is a different closure (holding a different value of i) created for each invocation of create_lambda.

5
  • The first approach doesn't work so well when you want your lambda to support *args. The second approach will work there, but is... more work :) Apr 9, 2012 at 10:30
  • 2
    Too bad I can't give the green mark to both answers. I chose this one because it actually gave the correct code to cut & paste like I'm five and I like that approach to answers on SO. Apr 9, 2012 at 16:10
  • For those of us less good at Python, any chance you could use a different variable than i for one of the instances in the first approach?
    – Chowlett
    Apr 9, 2012 at 19:24
  • 1
    @Chowlett: I prefer to use i in the example above because it is the common idiom to use the same variable name. You can also use [lambda k=i:k for i in range(10)] which would do the same thing.
    – interjay
    Apr 9, 2012 at 20:06
  • 2
    Cheers. If I were familiar with the syntax I think I'd do the same; but using k lets me see which letter is mapping to the in-lambda variable and which to the comprehension variable.
    – Chowlett
    Apr 9, 2012 at 20:24
11

What you're seeing here is the effect of closures. The lambda is capturing state from the program to be used later. So while each lambda is a unique object, the state isn't necessarily unique.

The actual 'gotchya' here, is that the variable i is captured, not the value that i represents at that point in time. We can illustrate this with a much easier example:

>>> y = 3
>>> f = lambda: y
>>> f()
3
>>> y = 4
>>> f()
4

The lambda holds on to the reference to the variable, and evaluates that variable when you execute the lambda.

To work around this, you can assign to a local variable within the lambda:

>>> f = lambda y=y:y
>>> f()
4
>>> y = 6
>>> f()
4

Finally, in the case of a loop, the loop variable is only 'declared' once. Therefore, any references to the loop variable within the loop will persist past the next iteration. This includes the variable in list comprehensions.

3
  • Well, any lambda function is a closure. I don't think that the poster was confused about what a closure was. The unexpected part is that it's using a variable reference in this case rather than the value. Most languages which use closures would substitute the value in this circumstance unless a reference type were used. Apr 9, 2012 at 8:11
  • 4
    @KeithIrwin, most languages that allow closures suffer from this exact 'gotchya'. Javascript and C# being two notable examples. Also, the OP never mentioned the word closure, so I figured I'd link to material that explains it if they were unaware. Apr 9, 2012 at 8:14
  • It shows up only in languages which don't clearly differentiate between values and references. For example, it shows up in the Lisp family of functional languages, but not in the more strongly-typed languages like the ML family or Haskell or Scala. If he'd been working in a language like F#, Ocaml, Haskell, or Scala, he would reasonably expect the exact opposite behavior. I assume that he has at least some experience with functional languages because he tagged the post as being about functional programming. Apr 9, 2012 at 8:40
1

The problem is that you're not capturing the value of i on each iteration of the list comprehension, you're capturing the variable each time through.

The problem is that a closure captures variables by reference. In this case you are capturing a variable whose value changes over time (as with all loop variables), so it has a different value when you run it than when you created it.

0

I'm not sure if it's a bug or a feature, but what's happening is that lambda:i doesn't evaluate i before forming the lambda function. So, it's literally just a function which evaluates whatever the current value of i is. Here's another example of how this happens.

>>> i=5
>>> x=lambda:i
>>> x()
5
>>> i=6
>>> x()
6

So, obviously, what's happening is the same thing except that i is going to 9 in your examples as it's being assigned through the range 0 through 9 in that order.

I don't think that there's really any good way to avoid it. Lambda functions in Python are pretty limited. It's not really a functional language at heart.

0

Turn a square bracket into a parenthesis, like that: [j() for j in (lambda:i for i in range(10))] output: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] Tuples are immutable variables

Not the answer you're looking for? Browse other questions tagged or ask your own question.