7

I have a list that contains many sentences. I want to iterate through the list, removing from all sentences words like "and", "the", "a", "are", etc.

I tried this:

def removearticles(text):


articles = {'a': '', 'an':'', 'and':'', 'the':''}
for i, j in articles.iteritems():
    text = text.replace(i, j)
return text

As you can probably tell, however, this will remove "a" and "an" when it appears in the middle of the word. I need to remove only the instances of the words when they are delimited by blank space, and not when they are within a word. What is the most efficient way of going about this?

5 Answers 5

9

I would go for regex, something like:

def removearticles(text):
  re.sub('(\s+)(a|an|and|the)(\s+)', '\1\3', text)

or if you want to remove the leading whitespace as well:

def removearticles(text):
  re.sub('\s+(a|an|and|the)(\s+)', '\2', text)
5
  • Brilliant! I knew somebody was going to submit regex as the solution. I'm afraid I'm an amateur, though. Do you know of any good regex tutorials/references? Jan 17, 2011 at 3:36
  • 1
    There's a tutorial here, I haven't done it all so I'm not sure how good it is though. The main tool I use for designing regexs is My Regex Tester, makes it easy to see where and why things match.
    – Nemo157
    Jan 17, 2011 at 3:38
  • Mastering Regular Expressions (by Friedl,O'reilly pub) is worth owning. amazon.com/Mastering-Regular-Expressions-Jeffrey-Friedl/dp/… Jan 17, 2011 at 3:40
  • doesnt work for me at all. >>> text 'my friend is a guy' >>> print re.sub('\s+(a|an|and|the)(\s+)', '\2', text) my friend isguy >>> print re.sub('\s+(a|an|and|the)(\s+)', '\1\3', text) my friend isguy >>> text = "a guy" >>> print re.sub('\s+(a|an|and|the)(\s+)', '\1\3', text) a guy >>> print re.sub('\s+(a|an|and|the)(\s+)', '\2', text) a guy Aug 26, 2015 at 18:25
  • Yeah, the second one doesn't work. Doesn't remove the spaces. Feb 1 at 18:15
4

This looks more like an NLP job than something you would do with straight regex. I would check out NLTK (http://www.nltk.org/) IIRC it comes with a corpus full of filler words like the ones you're trying to get rid of.

1
  • It's called "stopwords" or "Stopwords Corpus" and can be retrieved directly here. The full list of corpora is here. Jun 17, 2022 at 15:45
1
def removearticles(text):


articles = {'a': '', 'an':'', 'and':'', 'the':''}
rest = []
for word in text.split():
    if word not in articles:
        rest.append(word)
return ' '.join(rest)

in operator of dict run faster than list.

2
  • Does this run faster than regex? Jan 17, 2011 at 3:43
  • @ParseItongue I just test my method and regex solution provided by Senthil Kumaran. With Timer module, I process the NEWS.txt installed with Python2.6 which is 118kb. Both method run 1000 times, my method is a bit faster, around 2 seconds in my PC, than regex. Both method method is good enough, take anyone you like:)
    – xiaowl
    Jan 17, 2011 at 4:15
1

Try something along the lines of

articles = ['and', 'a']
newText = ''
for word in text.split(' '):
    if word not in articles:
        newText += word+' '
return newText[:-1]
1
  • Given that I'm unfamiliar with regex, I was thinking of solving the problem in the same way! Thank you. Jan 17, 2011 at 3:37
0

It can be done using regex. Iterator through your strings or (''.join the list and send it as a string) to the following regex.

>>> import re
>>> rx = re.compile(r'\ban\b|\bthe\b|\band\b|\ba\b')
>>> rx.sub(' ','a line with lots of an the and a baad')
'  line with lots of         baad'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.