0

I am trying to learn Python, and a project I've been assigned requires me to construct a script that can clear all punctuation from a text. This will later be used to count each word's frequency and convert it into a wordcloud. When trying to apply my script to an excerpt of the text (in this case Pride and Prejudice) the quotation marks seem to be skipped over even though they are included in my punctuation variable. What am I missing?

text = '''      “Already arisen?” repeated Mr. Bennet. “What, has she frightened
      away some of your lovers? Poor little Lizzy! But do not be cast
      down. Such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret. Come, let me see the
      list of pitiful fellows who have been kept aloof by Lydia’s
      folly.”'''
punctuation = '''!()-[]{};:'"\,<>./?@#$%^&*_~'''
text = text.lower()
clean = ""

for letter in text:
    if letter not in punctuation:
        clean += letter
print(clean)

As a result, I am returned the text with everything lowercase and without any punctuation except the quotation marks.

'''   “already arisen” repeated mr bennet “what has she frightened
      away some of your lovers poor little lizzy but do not be cast
      down such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret come let me see the
      list of pitiful fellows who have been kept aloof by lydia’s
      folly”'''

Please advise.

1 Answer 1

1

and are "smart quotes", not straight quotes ("). It's a completely different character. You can add them to your punctuation string if you like though.

To match and remove all Unicode forms of punctuation, you can use the technique from this answer, using the third party regex module:

import regex as re

def remove_punctuation(text):
    return re.sub(r"\p{P}+", "", text)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.