40

I'm using NLTK to analyze a few classic texts and I'm running in to trouble tokenizing the text by sentence. For example, here's what I get for a snippet from Moby Dick:

import nltk
sent_tokenize = nltk.data.load('tokenizers/punkt/english.pickle')

'''
(Chapter 16)
A clam for supper? a cold clam; is THAT what you mean, Mrs. Hussey?" says I, "but
that's a rather cold and clammy reception in the winter time, ain't it, Mrs. Hussey?"
'''
sample = 'A clam for supper? a cold clam; is THAT what you mean, Mrs. Hussey?" says I, "but that\'s a rather cold and clammy reception in the winter time, ain\'t it, Mrs. Hussey?"'

print "\n-----\n".join(sent_tokenize.tokenize(sample))
'''
OUTPUT
"A clam for supper?
-----
a cold clam; is THAT what you mean, Mrs.
-----
Hussey?
-----
" says I, "but that\'s a rather cold and clammy reception in the winter time, ain\'t it, Mrs.
-----
Hussey?
-----
"
'''

I don't expect perfection here, considering that Melville's syntax is a bit dated, but NLTK ought to be able to handle terminal double quotes and titles like "Mrs." Since the tokenizer is the result of an unsupervised training algo, however, I can't figure out how to tinker with it.

Anyone have recommendations for a better sentence tokenizer? I'd prefer a simple heuristic that I can hack rather than having to train my own parser.

4 Answers 4

53

You need to supply a list of abbreviations to the tokenizer, like so:

from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
punkt_param = PunktParameters()
punkt_param.abbrev_types = set(['dr', 'vs', 'mr', 'mrs', 'prof', 'inc'])
sentence_splitter = PunktSentenceTokenizer(punkt_param)
text = "is THAT what you mean, Mrs. Hussey?"
sentences = sentence_splitter.tokenize(text)

sentences is now:

['is THAT what you mean, Mrs. Hussey?']

Update: This does not work if the last word of the sentence has an apostrophe or a quotation mark attached to it (like Hussey?'). So a quick-and-dirty way around this is to put spaces in front of apostrophes and quotes that follow sentence-end symbols (.!?):

text = text.replace('?"', '? "').replace('!"', '! "').replace('."', '. "')
5
  • 1
    Ah, good to know. Strangely, this does not work if I run the complete sentence in my question through your solution. Any idea why? Dec 31, 2012 at 16:21
  • Just added some more info into the answer.
    – vpekar
    Jan 1, 2013 at 10:05
  • 5
    I generally avoid 'thanks' comments, but here it really is at place: thanks!
    – Private
    Apr 13, 2015 at 9:52
  • 1
    How do you handle the special case where the sentence has an apostrophe but you want to get the offsets? i.e. using span_tokenize method . The suggested workaround changes the original offsets.
    – CentAu
    May 19, 2015 at 15:08
  • 4
    The problem with this answer is it doesn't "tweak" the existing English tokenizer. You're going to lose a lot of the other features you might want if you create one from scratch. See stackoverflow.com/a/25375857/4582054
    – Josh Morel
    Jun 25, 2016 at 15:06
41

You can modify the NLTK's pre-trained English sentence tokenizer to recognize more abbreviations by adding them to the set _params.abbrev_types. For example:

extra_abbreviations = ['dr', 'vs', 'mr', 'mrs', 'prof', 'inc', 'i.e']
sentence_tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
sentence_tokenizer._params.abbrev_types.update(extra_abbreviations)

Note that the abbreviations must be specified without the final period, but do include any internal periods, as in 'i.e' above. For details about the other tokenizer parameters, refer to the relevant documentation.

5
  • 3
    This should be the top answer. If you just create a new tokenizer you won't get all the existing features of the English tokenizer.
    – Josh Morel
    Jun 25, 2016 at 15:05
  • 2
    It didn't seem to work for me, whereas the top answer did.
    – Alter
    Jun 27, 2016 at 18:14
  • @Alter You have to use it like so: sentence_tokenizer.tokenize(text) Jun 21, 2019 at 8:04
  • This also works for pickled customized tokenizers where you don't have to re-train it. (for languages that Punkt does not support) Feb 8, 2020 at 12:35
  • Be aware that nltk.data.load() uses a cache by default. So modifying sentence_tokenizer._params will also affect methods like nltk.sent_tokenize(). You can pass cache=False to avoid this. Apr 26, 2022 at 9:10
10

You can tell the PunktSentenceTokenizer.tokenize method to include "terminal" double quotes with the rest of the sentence by setting the realign_boundaries parameter to True. See the code below for an example.

I do not know a clean way to prevent text like Mrs. Hussey from being split into two sentences. However, here is a hack which

  • mangles all occurrences of Mrs. Hussey to Mrs._Hussey,
  • then splits the text into sentences with sent_tokenize.tokenize,
  • then for each sentence, unmangles Mrs._Hussey back to Mrs. Hussey

I wish I knew a better way, but this might work in a pinch.


import nltk
import re
import functools

mangle = functools.partial(re.sub, r'([MD]rs?[.]) ([A-Z])', r'\1_\2')
unmangle = functools.partial(re.sub, r'([MD]rs?[.])_([A-Z])', r'\1 \2')

sent_tokenize = nltk.data.load('tokenizers/punkt/english.pickle')

sample = '''"A clam for supper? a cold clam; is THAT what you mean, Mrs. Hussey?" says I, "but that\'s a rather cold and clammy reception in the winter time, ain\'t it, Mrs. Hussey?"'''    

sample = mangle(sample)
sentences = [unmangle(sent) for sent in sent_tokenize.tokenize(
    sample, realign_boundaries = True)]    

print u"\n-----\n".join(sentences)

yields

"A clam for supper?
-----
a cold clam; is THAT what you mean, Mrs. Hussey?"
-----
says I, "but that's a rather cold and clammy reception in the winter time, ain't it, Mrs. Hussey?"
1
  • Update: Consolidated part of this answer with the one above Jan 1, 2013 at 18:05
3

So I had a similar issue and tried out vpekar's solution above.

Perhaps mine is some sort of edge case but I observed the same behavior after applying the replacements, however, when I tried replacing the punctuation with the quotations placed before them, I got the output I was looking for. Presumably lack of adherence to MLA is less important than retaining the original quote as a single sentence.

To be more clear:

text = text.replace('?"', '"?').replace('!"', '"!').replace('."', '".')

If MLA is important though you could always go back and reverse these changes wherever it counts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.