55

Is it possible to put a function in a data structure, without first giving it a name with def?

# This is the behaviour I want. Prints "hi".
def myprint(msg):
    print msg
f_list = [ myprint ]
f_list[0]('hi')
# The word "myprint" is never used again. Why litter the namespace with it?

The body of a lambda function is severely limited, so I can't use them.

Edit: For reference, this is more like the real-life code where I encountered the problem.

def handle_message( msg ):
    print msg
def handle_warning( msg ):
    global num_warnings, num_fatals
    num_warnings += 1
    if ( is_fatal( msg ) ):
        num_fatals += 1
handlers = (
    ( re.compile( '^<\w+> (.*)' ), handle_message ),
    ( re.compile( '^\*{3} (.*)' ), handle_warning ),
)
# There are really 10 or so handlers, of similar length.
# The regexps are uncomfortably separated from the handler bodies,
# and the code is unnecessarily long.

for line in open( "log" ):
    for ( regex, handler ) in handlers:
        m = regex.search( line )
        if ( m ): handler( m.group(1) )
11
  • 1
    No it's not. # The word "myprint" is never used again. Why litter the namespace with it? Why do you invest so much time just to get rid of a single line that doesn't hurt you in any way?
    – phant0m
    Jul 8, 2011 at 20:13
  • 2
    @phant0m, @Udi: I want my code to be pretty and easy to read. In real life, I've got a list of pairs of regexp-function pairs and run the function/handler on strings that match the regexp. The handlers are small enough to make definitions outside the list ugly and inappropriate.
    – Tim
    Jul 8, 2011 at 20:18
  • 1
    I've added the real problem now. I usually dislike doing so, since it makes the question much more specific. I'll probably learn more from posting it, but not future visitors who've found the question by title. (ping @phant0m)
    – Tim
    Jul 8, 2011 at 20:50
  • 1
    Those function names are good documentation. If you were to make them anonymous the reader of your code would have to spend more brain cycles understanding what those functions do. Jul 8, 2011 at 21:22
  • 1
    If you really want to these things, you maybe want to switch to perl. I know perl, but I use python for clarity's sake. You could model this Pattern properly, or hack it. In the latter case I dont think, tat the namespace pollution is your primary problem. Jul 13, 2011 at 15:36

14 Answers 14

39

This is based on Udi's nice answer.

I think that the difficulty of creating anonymous functions is a bit of a red herring. What you really want to do is to keep related code together, and make the code neat. So I think decorators may work for you.

import re

# List of pairs (regexp, handler)
handlers = []

def handler_for(regexp):
    """Declare a function as handler for a regular expression."""
    def gethandler(f):
        handlers.append((re.compile(regexp), f))
        return f
    return gethandler

@handler_for(r'^<\w+> (.*)')
def handle_message(msg):
    print msg

@handler_for(r'^\*{3} (.*)')
def handle_warning(msg):
    global num_warnings, num_fatals
    num_warnings += 1
    if is_fatal(msg):
        num_fatals += 1
2
16

Nicer DRY way to solve your actual problem:

def message(msg):
    print msg
message.re = '^<\w+> (.*)'

def warning(msg):
    global num_warnings, num_fatals
    num_warnings += 1
    if ( is_fatal( msg ) ):
        num_fatals += 1
warning.re = '^\*{3} (.*)'

handlers = [(re.compile(x.re), x) for x in [
        message,
        warning,
        foo,
        bar,
        baz,
    ]]
1
  • Much nicer than my attempt. I really should read about the available data structures before coming up with more ideas. Thanks!
    – Tim
    Jul 8, 2011 at 21:33
15

Continuing Gareth's clean approach with a modular self contained solution:

import re

# in util.py
class GenericLogProcessor(object):

    def __init__(self):
      self.handlers = [] # List of pairs (regexp, handler)

    def register(self, regexp):
        """Declare a function as handler for a regular expression."""
        def gethandler(f):
            self.handlers.append((re.compile(regexp), f))
            return f
        return gethandler

    def process(self, file):
        """Process a file line by line and execute all handlers by registered regular expressions"""
        for line in file:
            for regex, handler in self.handlers:
                m = regex.search(line)
                if (m):
                  handler(m.group(1))      

# in log_processor.py
log_processor = GenericLogProcessor()

@log_processor.register(r'^<\w+> (.*)')
def handle_message(msg):
    print msg

@log_processor.register(r'^\*{3} (.*)')
def handle_warning(msg):
    global num_warnings, num_fatals
    num_warnings += 1
    if is_fatal(msg):
        num_fatals += 1

# in your code
with open("1.log") as f:
  log_processor.process(f)
2
  • I have to say, this one is nice and compact. Stuff comes together at one place. Jul 14, 2011 at 14:23
  • Excellent use of Decorators and keeping it all compact!
    – TyrantWave
    Jul 15, 2011 at 10:25
13

If you want to keep a clean namespace, use del:

def myprint(msg):
    print msg
f_list = [ myprint ]
del myprint
f_list[0]('hi')
9

As you stated, this can't be done. But you can approximate it.

def create_printer():
  def myprint(x):
    print x
  return myprint

x = create_printer()

myprint is effectively anonymous here, since the variable scope in which it was created is no longer accessible to the caller. (See closures in Python.)

1
  • I don't know how I feel about that last line. Isn't the entire point of closures in python that the scope persists inside the closure? That being said, I like this solution a lot.
    – Wilduck
    Jul 8, 2011 at 20:29
6

If you're concerned about polluting the namespace, create your functions inside of another function. Then you're only "polluting" the local namespace of the create_functions function and not the outer namespace.

def create_functions():
    def myprint(msg):
        print msg
    return [myprint]

f_list = create_functions()
f_list[0]('hi')
1
  • 2
    In addition to the namespace pollution, I'm also bothered by 1. using a temporary name and 2. having to define it in a different place when it's only going to be used once. Even though this solves the namespace problem, it only worsens the other problems.
    – Tim
    Jul 8, 2011 at 20:54
5

You should not do it cause eval is evil, but you can compile function code on run time using FunctionType and compile:

>>> def f(msg): print msg
>>> type(f)
 <type 'function'>
>>> help(type(f))
...
class function(object)
 |  function(code, globals[, name[, argdefs[, closure]]])
 |
 |  Create a function object from a code object and a dictionary.
 |  The optional name string overrides the name from the code object.
 |  The optional argdefs tuple specifies the default argument values.
 |  The optional closure tuple supplies the bindings for free variables.    
...

>>> help(compile)
Help on built-in function compile in module __builtin__:

compile(...)
    compile(source, filename, mode[, flags[, dont_inherit]]) -> code object

    Compile the source string (a Python module, statement or expression)
    into a code object that can be executed by the exec statement or eval().
    The filename will be used for run-time error messages.
    The mode must be 'exec' to compile a module, 'single' to compile a
    single (interactive) statement, or 'eval' to compile an expression.
    The flags argument, if present, controls which future statements influence
    the compilation of the code.
    The dont_inherit argument, if non-zero, stops the compilation inheriting
    the effects of any future statements in effect in the code calling
    compile; if absent or zero these statements do influence the compilation,
    in addition to any features explicitly specified.
3
  • Good idea. Out of interest, what is the moral difference between eval and compile?
    – Tim
    Jul 8, 2011 at 21:14
  • @Tim: stackoverflow.com/questions/2220699/… May 17, 2015 at 2:10
  • Fantastic answer. This novel approach is nothing less than a robust mechanism for dynamically defining Python callables (e.g., functions, methods) at runtime, complete with closure support! I've been grepping for something exactly like this for longer than I care to admit. Thanks for the putting the puzzle pieces together, Udi. Jun 25, 2016 at 7:08
3

The only way to make an anonymous function is with lambda, and as you know, they can only contain a single expression.

You can create a number of functions with the same name so at least you don't have to think of new names for each one.

It would be great to have truly anonymous functions, but Python's syntax can't support them easily.

5
  • 1
    "but Python's syntax can't support them easily." can you elaborate on this? (or link to resources) Thanks.
    – phant0m
    Jul 8, 2011 at 20:12
  • 3
    @phant0m: The model of indentation for block delimiting doesn't support starting/ending blocks inside expressions (e.g. inside parens), and it seems this can't be added easily (either very complex to implement or make indentation in all expression significant, which is breaks multi-line expressions). Therefore, you can't allow multiple statements in a lambda expression as you wouldn't know where is ended (no DEDENT token). I bet there's some more materal on the mailing list.
    – user395760
    Jul 8, 2011 at 20:21
  • Ah yes, that makes perfect sense! I didn't really think as much to get to think about where and how they would actually be used and what it then would mean :) Thanks!
    – phant0m
    Jul 8, 2011 at 20:27
  • 1
    I couldn't find a good description, but it has been rehashed many times, and @delnan is right: the indentation-based syntax can't accommodate statements in expressions. Jul 8, 2011 at 20:30
  • In contrast, the Scheme-derived R allows one to do: function(x,y,z) {w<- y+z ; x+w} where, for the uninitiated, "function" is the keyword that stands in for Python's "lambda". The main reason to use anonymous functions like this is to pass them as parameters to other functions.
    – user1544219
    Jul 22, 2015 at 22:29
3

As all said lambda is the only way, but you have to think not about lambda limitations but how to avoid them - for example you can use lists, dicts, comprehensions and so on in order to do what you want:

funcs = [lambda x,y: x+y, lambda x,y: x-y, lambda x,y: x*y, lambda x: x]
funcs[0](1,2)
>>> 3
funcs[1](funcs[0](1,2),funcs[0](2,2))
>>> -1
[func(x,y) for x,y in zip(xrange(10),xrange(10,20)) for func in funcs]

EDITED with print(try to look at the pprint module) and control-flow:

add = True
(funcs[0] if add else funcs[1])(1,2)
>>> 3

from pprint import pprint
printMsg = lambda isWarning, msg: pprint('WARNING: ' + msg) if isWarning else pprint('MSG:' + msg)
4
  • This seems quite involved, and mostly fit for mathematical expressions without flow control. Is it even possible to write a printer this way?
    – Tim
    Jul 8, 2011 at 20:21
  • I think that not only for mathematical, and flow control also can be used here - see my update Jul 8, 2011 at 20:59
  • I see, this might come in handy for some small work-arounds. Thanks.
    – Tim
    Jul 8, 2011 at 21:16
  • You are welcome, sorry if my solution doesn't fit your needs. I simply a beginner and experementing with python. Jul 8, 2011 at 21:19
3

Python really, really doesn't want to do this. Not only does it not have any way to define a multi-line anonymous function, but function definitions don't return the function, so even if this were syntactically valid...

mylist.sort(key=def _(v):
                    try:
                        return -v
                    except:
                        return None)

... it still wouldn't work. (Although I guess if it were syntactically valid, they'd make function definitions return the function, so it would work.)

So you can write your own function to make a function from a string (using exec of course) and pass in a triply-quoted string. It's kinda ugly syntactically, but it works:

def function(text, cache={}):

    # strip everything before the first paren in case it's "def foo(...):"
    if not text.startswith("("):
        text = text[text.index("("):]

    # keep a cache so we don't recompile the same func twice
    if text in cache:
        return cache[text]

    exec "def func" + text
    func.__name__ = "<anonymous>"

    cache[text] = func
    return func

    # never executed; forces func to be local (a tiny bit more speed)
    func = None

Usage:

mylist.sort(key=function("""(v):
                                try:
                                    return -v
                                except:
                                    return None"""))
3
  • Except for syntax highlightning, I don't find the custom function with triple-quote ugly at all. I didn't know about the func = None trick -- where is this documented?
    – Tim
    Jul 8, 2011 at 22:13
  • If you assign to a variable in a function, that variable is local. It's determined at compile time, not at execution, so it can be out of the actual path of execution. See: docs.python.org/reference/executionmodel.html, specifically "If a name is bound in a block, it is a local variable of that block."
    – kindall
    Jul 8, 2011 at 22:18
  • Speed advantage of local variables: wiki.python.org/moin/PythonSpeed/… I figured if I was gonna do something so hacky I'd at least make it as fast as possible. :-)
    – kindall
    Jul 8, 2011 at 22:21
2

Personally I'd just name it something use it and not fret about it "hanging around". The only thing you'll gain by using suggestions such as as redefining it later or using del to drop the name out of the namespace is a potential for confusion or bugs if someone later comes along and moves some code around without groking what you're doing.

1
  • In addition to the namespace litter, I'm also bothered by 1. using a temporary name and 2. having to define it in a different place when it's only going to be used once.
    – Tim
    Jul 8, 2011 at 20:53
2

You could use exec:

def define(arglist, body):
    g = {}
    exec("def anonfunc({0}):\n{1}".format(arglist,
                                     "\n".join("    {0}".format(line)
                                               for line in body.splitlines())), g)
    return g["anonfunc"]

f_list = [define("msg", "print(msg)")]
f_list[0]('hi')
2
  • This also works very well with multi-line code and indentation. Nice. The only exec overhead will be once, on the define call, right?
    – Tim
    Jul 8, 2011 at 21:13
  • Yes, the overhead is only once.
    – codeape
    Jul 9, 2011 at 9:22
1

The only option is to use a lambda expression, like you mention. Without that, it is not possible. That is the way python works.

4
  • In this case, or cases where I'd need two statements, lambda functions wouldn't work, right?
    – Tim
    Jul 8, 2011 at 20:10
  • Exactly. Since print isn't a statement it wont work. And you can only have one statement in a lambda. You can, however, make rather long statements with the and operator. But I guess that's not really what you're looking for. Jul 8, 2011 at 20:12
  • 1
    @thunderflower: you can't have any statements in a lambda, only an expression. Jul 8, 2011 at 20:31
  • Yeah. Sorry about that. Expression is what I meant. Jul 8, 2011 at 20:33
-1

If your function is complicated enough to not fit in a lambda function, then, for readability's sake, it would probably be best to define it in a normal block anyway.

4
  • Would you consider "print x" complicated?
    – Tim
    Jul 8, 2011 at 20:55
  • @Tim Nordenfur: He specifically said "The body of a lambda function is severely limited, so I can't use them." I assumed (perhaps incorrectly) that the print statement was an example. Jul 8, 2011 at 20:58
  • While there's surely a lot of too-long anonymous functions, I'm certain that there are some examples where a lambda-like syntax would be fine, even though it's too "complicated". print x is one counter-example, as is any statement that is not an expression.
    – Tim
    Jul 8, 2011 at 21:03
  • print can be used as a function by using from __future__ import print_statement. if statements can be handled by lambda. If he's needs to use something like a try...except, then defining a block for it is a good idea. Most other statements are either built in to lambda or useless to lambda function. Perhaps exec would work as a counter-example? Jul 8, 2011 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.