60

I am trying to remove stopwords from a string of text:

from nltk.corpus import stopwords
text = 'hello bye the the hi'
text = ' '.join([word for word in text.split() if word not in (stopwords.words('english'))])

I am processing 6 mil of such strings so speed is important. Profiling my code, the slowest part is the lines above, is there a better way to do this? I'm thinking of using something like regex's re.sub but I don't know how to write the pattern for a set of words. Can someone give me a hand and I'm also happy to hear other possibly faster methods.

Note: I tried someone's suggest of wrapping stopwords.words('english') with set() but that made no difference.

Thank you.

7
  • How large is stopwords.words('english')? Oct 24, 2013 at 8:24
  • @SteveBarnes A list of 127 words
    – mchangun
    Oct 24, 2013 at 8:26
  • 3
    did you wrap it inside list comprehension or outside? try add stw_set = set(stopwords.words('english')) and use this object instead
    – alko
    Oct 24, 2013 at 8:27
  • 1
    @alko I thought I wrapped it outside and had no effect, but I just tried it again and my code is running at least 10x faster now!!!
    – mchangun
    Oct 24, 2013 at 8:34
  • Are you processing the text line by line or all together?
    – Leonardo.Z
    Oct 24, 2013 at 8:39

6 Answers 6

120

Try caching the stopwords object, as shown below. Constructing this each time you call the function seems to be the bottleneck.

    from nltk.corpus import stopwords

    cachedStopWords = stopwords.words("english")

    def testFuncOld():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in stopwords.words("english")])

    def testFuncNew():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in cachedStopWords])

    if __name__ == "__main__":
        for i in xrange(10000):
            testFuncOld()
            testFuncNew()

I ran this through the profiler: python -m cProfile -s cumulative test.py. The relevant lines are posted below.

nCalls Cumulative Time

10000 7.723 words.py:7(testFuncOld)

10000 0.140 words.py:11(testFuncNew)

So, caching the stopwords instance gives a ~70x speedup.

4
  • Agreed. The performance boosts comes from caching the stopwords, not really in creating a set.
    – mchangun
    Oct 24, 2013 at 11:41
  • 12
    Certainly you get a dramatic boost from not having to read the list from disk every time, because that's the most time-consuming operation. But if you now turn your "cached" list into a set (just once, of course), you'll get another boost.
    – alexis
    May 9, 2015 at 13:47
  • can anyone tell me if this supports japanese? Mar 23, 2016 at 8:21
  • it gives me this UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal text=' '.join([word for word in text.split() if word not in stop_words]) please Salomone provide me solution to this Aug 16, 2016 at 14:43
31

Sorry for late reply. Would prove useful for new users.

  • Create a dictionary of stopwords using collections library
  • Use that dictionary for very fast search (time = O(1)) rather than doing it on list (time = O(stopwords))

    from collections import Counter
    stop_words = stopwords.words('english')
    stopwords_dict = Counter(stop_words)
    text = ' '.join([word for word in text.split() if word not in stopwords_dict])
    
6
  • This does indeed speed things up considerably even in comparison to regexp based approach.
    – Diego
    Sep 30, 2019 at 13:57
  • 1
    This was indeed a great answer and I wish this was more up there. It's incredible how fast this was when removing words from text from a list of 20k itens. Regular way took more than 1 hour, while using Counter took 20 seconds.
    – mrbTT
    Nov 22, 2019 at 16:28
  • Can you explain how 'Counter' speeds up the process? @Gulshan Jangid
    – Karan Bari
    Dec 14, 2019 at 13:32
  • 3
    well the main reason for the above code being fast is that we are searching in a dictionary which is basically a hashmap. And in hashmap the search time is O(1). Other than that Counter is part of collections library and library is written in C, and since C is way faster than python therefore Counter is faster than similar code written in python Dec 15, 2019 at 16:20
  • 3
    Using this (collections.Counter(stopwords.words('english')) cant be faster than using set(stopwords.words('english')) I believe. The collections.Counter method only unnecessarily uses more memory.
    – mikey
    Jun 16, 2021 at 12:56
25

Use a regexp to remove all words which do not match:

import re
pattern = re.compile(r'\b(' + r'|'.join(stopwords.words('english')) + r')\b\s*')
text = pattern.sub('', text)

This will probably be way faster than looping yourself, especially for large input strings.

If the last word in the text gets deleted by this, you may have trailing whitespace. I propose to handle this separately.

3
  • Any idea what the complexity of this would be? If w = number of words in my text and s = number of words in the stop list, I think looping would be on the order of w log s. In this case, w is approx s so it's w log w. Wouldn't grep be slower since it (roughly) has to match character by character?
    – mchangun
    Oct 24, 2013 at 11:40
  • 3
    Actually I think the complexities in the meaning of O(…) are the same. Both are O(w log s), yes. BUT regexps are implemented on a much lower level and optimized heavily. Already the splitting of words will lead to copying everything, creating a list of strings, and the list itself, all that takes precious time.
    – Alfe
    Oct 24, 2013 at 12:08
  • This approach is much faster than splitting lines, word tokenizing, then checking each word in a stopwords set. Particularly for larger text inputs Aug 25, 2020 at 13:38
7

First, you're creating stop words for each string. Create it once. Set would be great here indeed.

forbidden_words = set(stopwords.words('english'))

Later, get rid of [] inside join. Use generator instead.

Replace

' '.join([x for x in ['a', 'b', 'c']])

with

' '.join(x for x in ['a', 'b', 'c'])

Next thing to deal with would be to make .split() yield values instead of returning an array. I believe regex would be good replacement here. See thist hread for why s.split() is actually fast.

Lastly, do such a job in parallel (removing stop words in 6m strings). That is a whole different topic.

5
  • 1
    I doubt using regexp gonna be an improvement, see stackoverflow.com/questions/7501609/python-re-split-vs-split/…
    – alko
    Oct 24, 2013 at 8:34
  • Found it just now as well. :) Oct 24, 2013 at 8:38
  • 1
    Thanks. The set made at least an 8x improvement to speed. Why does using a generator help? RAM isn't an issue for me because each piece of text is quite small, about 100-200 words.
    – mchangun
    Oct 24, 2013 at 8:38
  • 2
    Actually, I've seen join perform better with a list comprehension than the equivalent generator expression. Oct 24, 2013 at 8:42
  • 1
    Set difference seems to work too clean_text = set(text.lower().split()) - set(stopwords.words('english'))
    – wmik
    Oct 25, 2019 at 15:50
2

Try using this by avoid looping and instead using regex to remove stopwords:

import re
from nltk.corpus import stopwords

cachedStopWords = stopwords.words("english")
pattern = re.compile(r'\b(' + r'|'.join(cachedStopwords) + r')\b\s*')
text = pattern.sub('', text)
0

Using just a regular dict seems to be the fastest solution by far.
Surpassing even the Counter solution by about 10%

from nltk.corpus import stopwords
stopwords_dict = {word: 1 for word in stopwords.words("english")}
text = 'hello bye the the hi'
text = " ".join([word for word in text.split() if word not in stopwords_dict])

Tested using the cProfile profiler

You can find the test code used here: https://gist.github.com/maxandron/3c276924242e7d29d9cf980da0a8a682

EDIT:

On top of that if we replace the list comprehension with a loop we get another 20% increase in performance

from nltk.corpus import stopwords
stopwords_dict = {word: 1 for word in stopwords.words("english")}
text = 'hello bye the the hi'

new = ""
for word in text.split():
    if word not in stopwords_dict:
        new += word
text = new

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.