26

I have some source code written for Python 3.5 that I want to make executable under Python 3.4. The only feature from 3.5 that I use which is not available in 3.4 are type hints, so I'd like to write a script to remove those entirely.

This seemed easy enough at first glance and I decided to write some regexes to do this, but then I thought about a few edge cases and I wasn't sure how to solve the problem for a more complex function like this:

def foo(bar: Dict[T, List[T]],
        baz: Callable[[T], int] = lambda x: (x+3)/7,
        **kwargs) -> List[T]:

Basically, I'd have to parse the whole thing and rebuild the argument list without type annotations. How would I even approach this?

6
  • 1
    The AST module is your friend in source transformation tasks, esp. the NodeTransformer. For the reverse you might need some 3rd party package. Mar 11, 2017 at 10:12
  • thanks. wow, that stuff is complicated... I wouldn't even know how to recognize type hints using this grammar
    – klamann
    Mar 11, 2017 at 10:32
  • 1
    Take a look at the grammar: for example args contain optional expr annotation and a FunctionDef optional expr returns. Your transformer would remove those. Mar 11, 2017 at 10:53
  • 4
    Btw. instead of removing the hints you could also use the backport module. But huge kudos for your AST solution! Mar 11, 2017 at 17:09
  • 1
    A minor nitpick about the original type hint for baz: the lambda uses true division and so it returns a float instead of an int. Mar 12, 2017 at 5:28

3 Answers 3

27

OK, I got it :D

Use Python's builtin ast module to parse the source code and then the excellent astunparse library to generate source code from the parsed ast again. Then all that's left is to remove the type annotations:

import ast
import astunparse

source="""
import typing
from typing import Dict, T, Callable
from typing import List

def foo(bar: Dict[T, List[T]],
        baz: Callable[[T], int] = lambda x: (x+3)/7,
        **kwargs) -> List[T]:
    pass
"""

class TypeHintRemover(ast.NodeTransformer):

    def visit_FunctionDef(self, node):
        # remove the return type defintion
        node.returns = None
        # remove all argument annotations
        if node.args.args:
            for arg in node.args.args:
                arg.annotation = None
        return node

    def visit_Import(self, node):
        node.names = [n for n in node.names if n.name != 'typing']
        return node if node.names else None

    def visit_ImportFrom(self, node):
        return node if node.module != 'typing' else None

# parse the source code into an AST
parsed_source = ast.parse(source)
# remove all type annotations, function return type definitions
# and import statements from 'typing'
transformed = TypeHintRemover().visit(parsed_source)
# convert the AST back to source code
print(astunparse.unparse(transformed))

The TypeHintRemover visits all Nodes in the AST and removes all type hints within function arguments, the return type definitions of each function and all import statements that refer to the 'typing' module.

The result is:

def foo(bar, baz=(lambda x: ((x + 3) / 7)), **kwargs):
    pass
6
  • what do you mean?
    – klamann
    Mar 11, 2017 at 15:37
  • you should read about type hints in Python 3, I'm a huge fan of those. Input and output are equivalent in this example when it comes to execution. Example: bar is supposed to be a Callable, which is just a type hint, indicated by :, and the lambda ist the default value of that argument. The type hint is completely optional and will not be evaluated during runtime.
    – klamann
    Mar 11, 2017 at 21:36
  • 1
    then please tell me what exactly is missing, because from what I can see, everything that is not a type hint is still there.
    – klamann
    Mar 11, 2017 at 22:44
  • 6
    Nothing is missing. I am going to hang my head in shame and delete this discussion thread. After finally getting a good night's sleep, I now realize that I simply can't read. Mar 13, 2017 at 15:04
  • Since you mentioned in the comments that you can't install any dependencies on the target platform, your answer of using the third-party astunparse module would not actually be ideal. You can use the built-in module lib2to3 instead as demonstrated in my answer.
    – blhsing
    Oct 4, 2019 at 17:07
6

There are also type hints for local variables (which came from Python 3.6). I've modified @klamann 's code to remove them too. Also, I use astor (https://pypi.org/project/astor/) to generate code.

import ast
import astor
import sys


class TypeHintRemover(ast.NodeTransformer):

    def visit_FunctionDef(self, node):
        # remove the return type definition
        node.returns = None
        # remove all argument annotations
        if node.args.args:
            for arg in node.args.args:
                arg.annotation = None
        self.generic_visit(node)
        return node

    def visit_AnnAssign(self, node):
        if node.value is None:
            return None
        return ast.Assign([node.target], node.value)

    def visit_Import(self, node):
        node.names = [n for n in node.names if n.name != 'typing']
        return node if node.names else None

    def visit_ImportFrom(self, node):
        return node if node.module != 'typing' else None

def remove_type_hints(source: str):
    # parse the source code into an AST
    parsed_source = ast.parse(source)
    # remove all type annotations, function return type definitions
    # and import statements from 'typing'
    transformed = TypeHintRemover().visit(parsed_source)
    # convert the AST back to source code
    return astor.to_source(transformed)


def main():
    _, source_name, dest_name = sys.argv
    with open(source_name, "r") as sourceFile:
        source = "\n".join(sourceFile.readlines())
        dest = remove_type_hints(source)
        with open(dest_name, "w") as destFile:
            destFile.write(dest)

if __name__ == "__main__":
    main()
1
  • Looks nice. Some improvements: Sometimes imports from typing are used in an isinstance call. Type aliases should be removed as well. It adds unneeded newlines in multiline docstrings. Also removes all comments, but don't know if you can get those back with this approach. Dec 23, 2020 at 13:45
3

You can subclass lib2to3.refactor.RefactoringTool to refactor the code using a fixer that is a subclass of lib2to3.fixer_base.BaseFix with a pattern that looks for either a typed argument, a function declaration with an annotated returning value, or a simple statement that imports or imports from typing, and a transform method that removes the indices of the annotations from the child nodes or replaces the statement node with an empty node:

from lib2to3 import fixer_base, refactor, fixer_util

class FixParameterAnnotations(fixer_base.BaseFix):
    PATTERN = r'''
        name=tname
        |
        func=funcdef< any+ '->' any+ >
        |
        simple_stmt<
            (
                import_name< 'import' 'typing' >
                |
                import_from< 'from' 'typing' 'import' any+ >
            ) '\n'
        >
    '''

    def transform(self, node, results):
        if 'name' in results:
            del node.children[1:] # delete annotation to typed argument
        elif 'func' in results:
            del node.children[-4:-2] # delete annotation to function declaration
        else:
            return fixer_util.BlankLine() # delete statement that imports typing
        return node

class Refactor(refactor.RefactoringTool):
    def __init__(self, fixers):
        self._fixers= [cls(None, None) for cls in fixers]
        super().__init__(None, {'print_function': True})

    def get_fixers(self):
        return self._fixers, []

so that:

source = """
import typing
from typing import Dict, T, Callable
from typing import List

def foo(bar: Dict[T, List[T]],
        baz: Callable[[T], int] = lambda x: (x+3)/7,
        **kwargs) -> List[T]:
    print(line, end="")    # comments and white spaces are preserved
"""
print(Refactor([FixParameterAnnotations]).refactor_string(source, ''))

outputs:

def foo(bar,
        baz = lambda x: (x+3)/7,
        **kwargs):
    print(line, end="")    # comments and white spaces are preserved

Demo: https://repl.it/@blhsing/BurlywoodFeistyTrials

As a bonus, lib2to3 also preserves all comments and white spaces after the transformation. You can find the definition of the Python grammar in Grammar.txt of the lib2to3 module.

8
  • that's an interesting solution, I wasn't aware that you could use lib2to3 like that. Is there any documentation on the language that is used to define the PATTERN? I'd like to learn more about it, but I wasn't able to find it in the lib2to3 docs.
    – klamann
    Oct 5, 2019 at 8:01
  • Although this idea is really interesting, 2to3 is not designed to process python3 code (and the code with annotations is definitely python3). It crashes even on simple cases like print(line, end="")
    – Ark-kun
    Feb 19, 2020 at 19:55
  • @Ark-kun This can be fixed by enabling the print_function option when initializing refactor.RefactoringTool. I've updated my answer accordingly.
    – blhsing
    Feb 19, 2020 at 20:55
  • @klamann Unfortunately there is no good official documentation for lib2to3 so I learned it by reading the code myself. I do now find that there are some good articles about it after Googling it. Here is one of the articles that I think you will find helpful: python3porting.com/fixers.html
    – blhsing
    Feb 19, 2020 at 20:59
  • @blhsing Thanks for the quick fix! Since you know a lot about lib2to3, I'd like to ask: how would you access the probability of some similar issue remaining there? I have a feeling that i should probably use an tokenize/ast-based solution for prod.
    – Ark-kun
    Feb 19, 2020 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.