116

I was trying to remove unwanted characters from a given string using text.translate() in Python 3.4.

The minimal code is:

import sys 
s = 'abcde12345@#@$#%$'
mapper = dict.fromkeys(i for i in range(sys.maxunicode) if chr(i) in '@#$')
print(s.translate(mapper))

It works as expected. However the same program when executed in Python 3.4 and Python 3.5 gives a large difference.

The code to calculate timings is

python3 -m timeit -s "import sys;s = 'abcde12345@#@$#%$'*1000 ; mapper = dict.fromkeys(i for i in range(sys.maxunicode) if chr(i) in '@#$'); "   "s.translate(mapper)"

The Python 3.4 program takes 1.3ms whereas the same program in Python 3.5 takes only 26.4μs.

What has improved in Python 3.5 that makes it faster compared to Python 3.4?

5
  • 11
    While we're talking about performance, wouldn't it be better to generate your mapper like this: dict.fromkeys(ord(c) for c in '@#$')?
    – Thomas K
    Dec 15, 2015 at 11:49
  • 1
    @ThomasK I found out that this made a significant difference. Yep your way is better. Dec 15, 2015 at 12:06
  • Did you mean 50x faster?
    – assylias
    Dec 18, 2015 at 9:02
  • @assylias I did 1300 - 26.4 and then divided by 1300. I got nearly 95%, so I wrote :) It is actually more than 50x faster... But is my calculation wrong? I'm bit weak in math. I'll learn math soon. :) Dec 18, 2015 at 9:17
  • 3
    you should do it the way round: 26 / 1300 = 2% so the faster version takes only 2% of the time taken by the slower version => it is 50x faster.
    – assylias
    Dec 18, 2015 at 9:20

1 Answer 1

152

TL;DR - ISSUE 21118


The long Story

Josh Rosenberg found out that the str.translate() function is very slow compared to the bytes.translate, he raised an issue, stating that:

In Python 3, str.translate() is usually a performance pessimization, not optimization.

Why was str.translate() slow?

The main reason for str.translate() to be very slow was that the lookup used to be in a Python dictionary.

The usage of maketrans made this problem worse. The similar approach using bytes builds a C array of 256 items to fast table lookup. Hence the usage of higher level Python dict makes the str.translate() in Python 3.4 very slow.

What happened now?

The first approach was to add a small patch, translate_writer, However the speed increase was not that pleasing. Soon another patch fast_translate was tested and it yielded very nice results of up to 55% speedup.

The main change as can be seen from the file is that the Python dictionary lookup is changed into a C level lookup.

The speeds now are almost the same as bytes

                                unpatched           patched

str.translate                   4.55125927699919    0.7898181750006188
str.translate from bytes trans  1.8910855210015143  0.779950579000797

A small note here is that the performance enhancement is only prominent in ASCII strings.

As J.F.Sebastian mentions in a comment below, Before 3.5, translate used to work in the same way for both ASCII and non-ASCII cases. However from 3.5 ASCII case is much faster.

Earlier ASCII vs non-ascii used to be almost same, however now we can see a great change in the performance.

It can be an improvement from 71.6μs to 2.33μs as seen in this answer.

The following code demonstrates this

python3.5 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)"
100000 loops, best of 3: 2.3 usec per loop
python3.5 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)"
10000 loops, best of 3: 117 usec per loop

python3 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)"
10000 loops, best of 3: 91.2 usec per loop
python3 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)"
10000 loops, best of 3: 101 usec per loop

Tabulation of the results:

         Python 3.4    Python 3.5  
Ascii     91.2          2.3 
Unicode   101           117
12
  • 13
    This is one of the commits: github.com/python/cpython/commit/…
    – filmor
    Dec 15, 2015 at 11:40
  • note: ascii vs. non-ascii case may differ significantly in performance. It is not about 55%: as your answer shows, the speed up can be 1000s%.
    – jfs
    Dec 15, 2015 at 19:51
  • compare: python3.5 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)" (ascii) vs. python3.5 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)" (non-ascii). The latter is much (10x) slower.
    – jfs
    Dec 15, 2015 at 20:07
  • @J.F. Oh, I understood it now. I ran your code for both 3.4 and 3.5. I am getting Py3.4 faster for non-ascii stuff. Is it by coincidence? The results dpaste.com/15FKSDQ Dec 15, 2015 at 20:14
  • Before 3.5, both ascii and non-ascii cases are probably the same for Unicode .translate() i.e., ascii case is much faster in Python 3.5 only (you don't need bytes.translate() for performance there).
    – jfs
    Dec 15, 2015 at 20:20

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.