172

Function Annotations: PEP-3107

I ran across a snippet of code demonstrating Python3's function annotations. The concept is simple but I can't think of why these were implemented in Python3 or any good uses for them. Perhaps SO can enlighten me?

How it works:

def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
    ... function body ...

Everything following the colon after an argument is an 'annotation', and the information following the -> is an annotation for the function's return value.

foo.func_annotations would return a dictionary:

{'a': 'x',
 'b': 11,
 'c': list,
 'return': 9}

What's the significance of having this available?

5
  • 15
  • 6
    @SilentGhost: unfortunately, many of the links with the actual use cases are broken. Is there any place where the content might have been stored, or it's gone forever?
    – max
    Aug 24, 2012 at 17:53
  • 17
    shouldn't foo.func_annotations be foo.__annotations__ in python3 ? Feb 26, 2014 at 5:57
  • 3
    Annotations have no special significance. The only thing that Python does is to put them in the annotations dictionary. Any other action is up to you.
    – N Randhawa
    Aug 21, 2016 at 11:45
  • 1
    what does def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9): mean?
    – Ali SH
    Aug 3, 2018 at 6:16

12 Answers 12

102

Function annotations are what you make of them.

They can be used for documentation:

def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second'):
     ...

They can be used for pre-condition checking:

def validate(func, locals):
    for var, test in func.__annotations__.items():
        value = locals[var]
        msg = 'Var: {0}\tValue: {1}\tTest: {2.__name__}'.format(var, value, test)
        assert test(value), msg


def is_int(x):
    return isinstance(x, int)

def between(lo, hi):
    def _between(x):
            return lo <= x <= hi
    return _between

def f(x: between(3, 10), y: is_int):
    validate(f, locals())
    print(x, y)


>>> f(0, 31.1)
Traceback (most recent call last):
   ... 
AssertionError: Var: y  Value: 31.1 Test: is_int

Also see http://www.python.org/dev/peps/pep-0362/ for a way to implement type checking.

7
  • 23
    How is this better than a docstring for documentation, or explicit type checking in the function? This seems to complicate the language for no reason.
    – endolith
    Apr 23, 2013 at 20:34
  • 11
    @endolith We can certainly do without function annotations. They just provide standard way to access the annotations. That makes them accessible to help() and to tool-tips and makes them available for introspection. Apr 23, 2013 at 21:12
  • 4
    Rather than passing around numbers you could create types Mass and Velocity instead.
    – user1804599
    Jul 19, 2014 at 11:43
  • 1
    to fully demonstrate this I would have def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second') -> float: to also show the return type. This is my favorite answer on here.
    – Tommy
    Jan 15, 2016 at 0:11
  • 1
    @user189728 You're correct. Either the return value needs to be saved to a variable or the whole function needs to be wrapped in a validating decorator. Jul 12, 2018 at 7:38
95

I think this is actually great.

Coming from an academic background, I can tell you that annotations have proved themselves invaluable for enabling smart static analyzers for languages like Java. For instance, you could define semantics like state restrictions, threads that are allowed to access, architecture limitations, etc., and there are quite a few tools that can then read these and process them to provide assurances beyond what you get from the compilers. You could even write things that check preconditions/postconditions.

I feel something like this is especially needed in Python because of its weaker typing, but there were really no constructs that made this straightforward and part of the official syntax.

There are other uses for annotations beyond assurance. I can see how I could apply my Java-based tools to Python. For instance, I have a tool that lets you assign special warnings to methods, and gives you indications when you call them that you should read their documentation (E.g., imagine you have a method that must not be invoked with a negative value, but it's not intuitive from the name). With annotations, I could technically write something like this for Python. Similarly, a tool that organizes methods in a large class based on tags can be written if there is an official syntax.

7
  • 35
    ISTM these are theoretical benefits that can only be realized only if the standard library and third-party modules all use function annotations and use them with consistent meaning and use a well-thought out systems of annotations. Until that day (which will never come), the main uses of Python's function annotations will be the one-off uses described in the other answers. For the time being, you can forget about smart static analyzers, compiler assurances, java-based tool chains, etc. Jan 21, 2012 at 23:14
  • 6
    Even without everything using function annotations you can still use them for static analysis within code that has them on its inputs and is calling other code that is similarly annotated. Within a larger project or codebase this could still be a significantly useful body of code to perform annotation based static analysis on.
    – gps
    Apr 1, 2012 at 5:56
  • 1
    AFAICT, you can do all of this with decorators, which predate annotations; therefore, I still don't see the benefit. I have a slightly different take on this question: stackoverflow.com/questions/13784713/… Dec 9, 2012 at 4:52
  • 9
    Fast-forward to 2015 , python.org/dev/peps/pep-0484 and mypy-lang.org are starting to prove all naysayers wrong. Sep 23, 2015 at 10:37
  • 2
    @DustinWyatt I'm glad to have been wrong about that forecast :-) We did get standardized types from PEP 484 and a mostly annotated standard library with typeshed. However, the OP's wishlist for Java-style tooling mostly hasn't come to fruition yet. Sep 4, 2018 at 5:15
49

This is a way late answer, but AFAICT, the best current use of function annotations is PEP-0484 and MyPy. There's also PyRight from Microsoft which is used by VSCode and also available via CLI.

Mypy is an optional static type checker for Python. You can add type hints to your Python programs using the upcoming standard for type annotations introduced in Python 3.5 beta 1 (PEP 484), and use mypy to type check them statically.

Used like so:

from typing import Iterator

def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a + b
4
  • 2
    More examples here Mypy Examples and here How You Can Benefit from Type Hints
    – El Ruso
    Nov 6, 2015 at 20:11
  • Also see pytype - the other static analyzer being built with PEP-0484 in mind.
    – gps
    Jul 12, 2016 at 1:11
  • Unfortunately the type isn't enforced. If I type list(fib('a')) with your example function, Python 3.7 happily accepts the argument and complains about there being no way to compare a string and an int. Jun 13, 2019 at 18:48
  • @DenisdeBernardy As PEP-484 explains Python only provides type annotations. To enforce types you have to use mypy. Jun 14, 2019 at 21:25
25

Just to add a specific example of a good use from my answer here, coupled with decorators a simple mechanism for multimethods can be done.

# This is in the 'mm' module

registry = {}
import inspect

class MultiMethod(object):
    def __init__(self, name):
        self.name = name
        self.typemap = {}
    def __call__(self, *args):
        types = tuple(arg.__class__ for arg in args) # a generator expression!
        function = self.typemap.get(types)
        if function is None:
            raise TypeError("no match")
        return function(*args)
    def register(self, types, function):
        if types in self.typemap:
            raise TypeError("duplicate registration")
        self.typemap[types] = function

def multimethod(function):
    name = function.__name__
    mm = registry.get(name)
    if mm is None:
        mm = registry[name] = MultiMethod(name)
    spec = inspect.getfullargspec(function)
    types = tuple(spec.annotations[x] for x in spec.args)
    mm.register(types, function)
    return mm

and an example of use:

from mm import multimethod

@multimethod
def foo(a: int):
    return "an int"

@multimethod
def foo(a: int, b: str):
    return "an int and a string"

if __name__ == '__main__':
    print("foo(1,'a') = {}".format(foo(1,'a')))
    print("foo(7) = {}".format(foo(7)))

This can be done by adding the types to the decorator as Guido's original post shows, but annotating the parameters themselves is better as it avoids the possibility of wrong matching of parameters and types.

Note: In Python you can access the annotations as function.__annotations__ rather than function.func_annotations as the func_* style was removed on Python 3.

5
  • 2
    Interesting application, though I'm afraid function = self.typemap.get(types) won't work when subclasses are involved. In that case you'd probably have to loop through typemap using isinnstance. I wonder if @overload handles this correctly Aug 28, 2013 at 7:28
  • I think this is broken if the function has a return type
    – zenna
    Oct 16, 2016 at 16:21
  • 1
    The __annotations__ is a dict that does not ensure arguments order, so this snippet sometimes fail. I would recommend changing the types = tuple(...) to spec = inspect.getfullargspec(function) then types = tuple([spec.annotations[x] for x in spec.args]).
    – xoolive
    Dec 14, 2016 at 22:24
  • You are quite correct, @xoolive. Why don't you edit the answer to add your fix? Dec 15, 2016 at 12:11
  • @xoolive: I noticed. Sometimes the editors use a heavy hand in managing the edits. I have edited the question to include your fix. Actually, I have had a discussion about this, but there is no way to unreject the fix. Thanks for the help by the way. Dec 20, 2016 at 18:53
24

Uri has already given a proper answer, so here's a less serious one: So you can make your docstrings shorter.

6
  • 2
    love it. +1. however, in the end, writing docstrings is still the number one way I make my code readable , however, if you were to implement any kind of static or dynamic checking, it is nice to have this. Perhaps I might find a use for it.
    – Warren P
    Jul 9, 2010 at 20:37
  • 8
    I do not recommend using annotations as a replacement for an Args: section or @param lines or similar in your docstrings (whatever format you choose to use). While documentation annotations make for a pretty example, it tarnishes the potential power of annotations as it could get in the way of other more powerful uses. Also, you cannot omit annotations at runtime to reduce memory consumption (python -OO) as you can with docstrings and assert statements.
    – gps
    Apr 1, 2012 at 6:02
  • 2
    @gps: Like I said, it was a less serious answer.
    – JAB
    Apr 30, 2012 at 17:14
  • 3
    In all seriousness, it's a much better way to document types that you expect, while still adhereing to DuckTyping.
    – Marc
    Oct 23, 2015 at 22:44
  • 1
    @gps I'm not sure the memory consumption of docstrings is something to worry about in 99.999% of cases.
    – Tommy
    Jan 15, 2016 at 0:12
14

The first time I saw annotations, I thought "great! Finally I can opt in to some type checking!" Of course, I hadn't noticed that annotations are not actually enforced.

So I decided to write a simple function decorator to enforce them:

def ensure_annotations(f):
    from functools import wraps
    from inspect import getcallargs
    @wraps(f)
    def wrapper(*args, **kwargs):
        for arg, val in getcallargs(f, *args, **kwargs).items():
            if arg in f.__annotations__:
                templ = f.__annotations__[arg]
                msg = "Argument {arg} to {f} does not match annotation type {t}"
                Check(val).is_a(templ).or_raise(EnsureError, msg.format(arg=arg, f=f, t=templ))
        return_val = f(*args, **kwargs)
        if 'return' in f.__annotations__:
            templ = f.__annotations__['return']
            msg = "Return value of {f} does not match annotation type {t}"
            Check(return_val).is_a(templ).or_raise(EnsureError, msg.format(f=f, t=templ))
        return return_val
    return wrapper

@ensure_annotations
def f(x: int, y: float) -> float:
    return x+y

print(f(1, y=2.2))

>>> 3.2

print(f(1, y=2))

>>> ensure.EnsureError: Argument y to <function f at 0x109b7c710> does not match annotation type <class 'float'>

I added it to the Ensure library.

1
  • I have the same disappointment after I was exited believing Python finally at last has type checking. Will finally have to go‑on with house‑made type check implementation.
    – Hibou57
    Jul 19, 2014 at 19:58
3

It a long time since this was asked but the example snippet given in the question is (as stated there as well) from PEP 3107 and at the end of thas PEP example Use cases are also given which might answer the question from the PEPs point of view ;)

The following is quoted from PEP3107

Use Cases

In the course of discussing annotations, a number of use-cases have been raised. Some of these are presented here, grouped by what kind of information they convey. Also included are examples of existing products and packages that could make use of annotations.

  • Providing typing information
    • Type checking ([3], [4])
    • Let IDEs show what types a function expects and returns ([17])
    • Function overloading / generic functions ([22])
    • Foreign-language bridges ([18], [19])
    • Adaptation ([21], [20])
    • Predicate logic functions
    • Database query mapping
    • RPC parameter marshaling ([23])
  • Other information
    • Documentation for parameters and return values ([24])

See the PEP for more information on specific points (as well as their references)

1
  • I would really apprreciate if downvoters leave at least a short comment what caused the downvote. This would really help (at least me) a lot to improve.
    – klaas
    Jul 24, 2017 at 10:26
3

Python 3.X (only) also generalizes function definition to allow arguments and return values to be annotated with object values for use in extensions.

Its META-data to explain, to be more explicit about the function values.

Annotations are coded as :value after the argument name and before a default, and as ->value after the argument list.

They are collected into an __annotations__ attribute of the function, but are not otherwise treated as special by Python itself:

>>> def f(a:99, b:'spam'=None) -> float:
... print(a, b)
...
>>> f(88)
88 None
>>> f.__annotations__
{'a': 99, 'b': 'spam', 'return': <class 'float'>}

Source: Python Pocket Reference, Fifth Edition

EXAMPLE:

The typeannotations module provides a set of tools for type checking and type inference of Python code. It also a provides a set of types useful for annotating functions and objects.

These tools are mainly designed to be used by static analyzers such as linters, code completion libraries and IDEs. Additionally, decorators for making run-time checks are provided. Run-time type checking is not always a good idea in Python, but in some cases it can be very useful.

https://github.com/ceronman/typeannotations

How Typing Helps to Write Better Code

Typing can help you do static code analysis to catch type errors before you send your code to production and prevent you from some obvious bugs. There are tools like mypy, which you can add to your toolbox as part of your software life cycle. mypy can check for correct types by running against your codebase partially or fully. mypy also helps you to detect bugs such as checking for the None type when the value is returned from a function. Typing helps to make your code cleaner. Instead of documenting your code using comments, where you specify types in a docstring, you can use types without any performance cost.

Clean Python: Elegant Coding in Python ISBN: ISBN-13 (pbk): 978-1-4842-4877-5

PEP 526 -- Syntax for Variable Annotations

https://www.python.org/dev/peps/pep-0526/

https://www.attrs.org/en/stable/types.html

2
  • @BlackJack, the "for use in extensions" was not clear?
    – The Demz
    Dec 5, 2017 at 14:22
  • It is clear, but doesn't answer the question IMHO. It's like answering „What are good uses of classes?“ with „For use in programs.“ It's clear, correct, but the asking party is not really wiser as to what the heck good concrete uses are. Your's is an answer that can't be more generic, with an example that's essentially the same as the one already in the question.
    – BlackJack
    Dec 5, 2017 at 14:51
1

Despite all uses described here, the one enforceable and, most likely, enforced use of annotations will be for type hints.

This is currently not enforced in any way but, judging from PEP 484, future versions of Python will only allow types as the value for annotations.

Quoting What about existing uses of annotations?:

We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411 ) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8.

Though I haven't seen any silent deprecations in 3.6 yet, this could very well be bumped to 3.7, instead.

So, even though there might be some other good use-cases, it is best to keep them solely for type hinting if you don't want to go around changing everything in a future where this restriction is in place.

1

As a bit of a delayed answer, several of my packages (marrow.script, WebCore, etc.) use annotations where available to declare typecasting (i.e. transforming incoming values from the web, detecting which arguments are boolean switches, etc.) as well as to perform additional markup of arguments.

Marrow Script builds a complete command-line interface to arbitrary functions and classes and allows for defining documentation, casting, and callback-derived default values via annotations, with a decorator to support older runtimes. All of my libraries that use annotations support the forms:

any_string  # documentation
any_callable  # typecast / callback, not called if defaulting
(any_callable, any_string)  # combination
AnnotationClass()  # package-specific rich annotation object
[AnnotationClass(), AnnotationClass(), …]  # cooperative annotation

"Bare" support for docstrings or typecasting functions allows for easier mixing with other libraries that are annotation-aware. (I.e. have a web controller using typecasting that also happens to be exposed as a command-line script.)

Edited to add: I've also begun making use of the TypeGuard package using development-time assertions for validation. Benefit: when run with "optimizations" enabled (-O / PYTHONOPTIMIZE env var) the checks, which may be expensive (e.g. recursive) are omitted, with the idea that you've properly tested your app in development so the checks should be unnecessary in production.

-2

Annotations can be used for easily modularizing code. E.g. a module for a program which I'm maintaining could just define a method like:

def run(param1: int):
    """
    Does things.

    :param param1: Needed for counting.
    """
    pass

and we could ask the user for a thing named "param1" which is "Needed for counting" and should be an "int". In the end we can even convert the string given by the user to the desired type to get the most hassle free experience.

See our function metadata object for an open source class which helps with this and can automatically retrieve needed values and convert them to any desired type (because the annotation is a conversion method). Even IDEs show autocompletions right and assume that types are according to annotations - a perfect fit.

-2

If you look at the list of benefits of Cython, a major one is the ability to tell the compiler which type a Python object is.

I can envision a future where Cython (or similar tools that compile some of your Python code) will use the annotation syntax to do their magic.

1
  • The RPython Annotator is an example of an approach that feels suitably Pythonic; after generating a graph of your application, it can work out the type of every variable and (for RPython) enforce single-type safety. OTOH it requires "boxing" or other solutions/work-arounds to allow for dynamic rich values. Who am I to force my multiply function to only work against integers, when 'na' * 8 + ' batman!' is entirely valid? ;)
    – amcgregor
    Jan 19, 2019 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.