5

I try to do some process on a text. It's part of my code:

fp = open(train_file)
raw = fp.read()
sents = fp.readlines()
words = nltk.tokenize.word_tokenize(raw)
bigrams = ngrams(words,2, left_pad_symbol='<s>', right_pad_symbol=</s>)
fdist = nltk.FreqDist(words)

In the old versions of nltk I found this code on StackOverflow for perplexity

estimator = lambda fdist, bins: LidstoneProbDist(fdist, 0.2) 
lm = NgramModel(5, train, estimator=estimator)
print("len(corpus) = %s, len(vocabulary) = %s, len(train) = %s, len(test) = %s" % ( len(corpus), len(vocabulary), len(train), len(test) ))
print("perplexity(test) =", lm.perplexity(test))   

However, this code is no longer valid, and I didn't find any other package or function in nltk for this purpose. Should I implement it?

1 Answer 1

29

Perplexity

Lets assume we have a model which takes as input an English sentence and gives out a probability score corresponding to how likely its is a valid English sentence. We want to determined how good this model is. A good model should give high score to valid English sentences and low score to invalid English sentences. Perplexity is a popularly used measure to quantify how "good" such a model is. If a sentence s contains n words then perplexity

pic

Modeling probability distribution p (building the model)

pic can be expanded using chain rule of probability

pic

So given some data (called train data) we can calculated the above conditional probabilities. However, practically it is not possible as it will requires huge amount of training data. We then make assumption to calculate

pic

Assumption : All words are independent (unigram)

pic

Assumption : First order Markov assumption (bigram)

Next words depends only on the previous word

pic

Assumption : n order Markov assumption (ngram)

Next words depends only on the previous n words

MLE to estimate probabilities

Maximum Likelihood Estimate(MLE) is one way to estimate the individual probabilities

Unigram

pic where

  • count(w) is number of times the word w appears in the train data

  • count(vocab) is the number of uniques words (called vocabulary) in the train data.

Bigram

pic where

  • count(w_{i-1}, w_i) is number of times the words w_{i-1}, w_i appear together in same sequence (bigram) in the train data

  • count(w_{i-1}) is the number of times the word w_{i-1} appear in the train data. w_{i-1} is called context.

Calculating Perplexity

As we have seen above $p(s)$ is calculated by multiplying lots of small numbers and so it is not numerically stable because of limited precision of floating point numbers on a computer. Lets use the nice properties of log to simply it. We know pic

pic

Example: Unigram model

Train Data ["an apple", "an orange"] Vocabulary : [an, apple, orange, UNK]

MLE estimates

pic

For test sentence "an apple"

l =  (np.log2(0.5) + np.log2(0.25))/2 = -1.5
np.power(2, -l) = 2.8284271247461903

For test sentence "an ant"

l =  (np.log2(0.5) + np.log2(0))/2 = inf
Code
import nltk
from nltk.lm.preprocessing import padded_everygram_pipeline
from nltk.lm import MLE

train_sentences = ['an apple', 'an orange']
tokenized_text = [list(map(str.lower, nltk.tokenize.word_tokenize(sent))) 
                for sent in train_sentences]
n = 1
train_data, padded_vocab = padded_everygram_pipeline(n, tokenized_text)
model = MLE(n)
model.fit(train_data, padded_vocab)

test_sentences = ['an apple', 'an ant']
tokenized_text = [list(map(str.lower, nltk.tokenize.word_tokenize(sent))) 
                for sent in test_sentences]

test_data, _ = padded_everygram_pipeline(n, tokenized_text)
for test in test_data:
    print ("MLE Estimates:", [((ngram[-1], ngram[:-1]),model.score(ngram[-1], ngram[:-1])) for ngram in test])

test_data, _ = padded_everygram_pipeline(n, tokenized_text)

for i, test in enumerate(test_data):
  print("PP({0}):{1}".format(test_sentences[i], model.perplexity(test)))

Example: Bigram model

Train Data: "an apple", "an orange" Padded Train Data: "(s) an apple (/s)", "(s) an orange (/s)" Vocabulary : (s), (/s) an, apple, orange, UNK

MLE estimates

pic

For test sentence "an apple" Padded : "(s) an apple (/s)"

l =  (np.log2(p(an|<s> ) + np.log2(p(apple|an) + np.log2(p(</s>|apple))/3 = 
(np.log2(1) + np.log2(0.5) + np.log2(1))/3 = -0.3333
np.power(2, -l) = 1.

For test sentence "an ant" Padded : "(s) an ant (/s)"

l =  (np.log2(p(an|<s> ) + np.log2(p(ant|an) + np.log2(p(</s>|ant))/3 = inf
Code
import nltk
from nltk.lm.preprocessing import padded_everygram_pipeline
from nltk.lm import MLE
from nltk.lm import Vocabulary

train_sentences = ['an apple', 'an orange']
tokenized_text = [list(map(str.lower, nltk.tokenize.word_tokenize(sent))) for sent in train_sentences]

n = 2
train_data = [nltk.bigrams(t,  pad_right=True, pad_left=True, left_pad_symbol="<s>", right_pad_symbol="</s>") for t in tokenized_text]
words = [word for sent in tokenized_text for word in sent]
words.extend(["<s>", "</s>"])
padded_vocab = Vocabulary(words)
model = MLE(n)
model.fit(train_data, padded_vocab)

test_sentences = ['an apple', 'an ant']
tokenized_text = [list(map(str.lower, nltk.tokenize.word_tokenize(sent))) for sent in test_sentences]

test_data = [nltk.bigrams(t,  pad_right=True, pad_left=True, left_pad_symbol="<s>", right_pad_symbol="</s>") for t in tokenized_text]
for test in test_data:
    print ("MLE Estimates:", [((ngram[-1], ngram[:-1]),model.score(ngram[-1], ngram[:-1])) for ngram in test])

test_data = [nltk.bigrams(t,  pad_right=True, pad_left=True, left_pad_symbol="<s>", right_pad_symbol="</s>") for t in tokenized_text]
for i, test in enumerate(test_data):
  print("PP({0}):{1}".format(test_sentences[i], model.perplexity(test)))
3
  • @mujjiga, so in a bigram model and higher the Vocabulary play no roll in calculating the probability? Commented Apr 20, 2020 at 10:26
  • Do you know whether this perplexity is computed with smoothing? What will happen if in the test text exists not seen word? Commented May 26, 2021 at 11:55
  • 2
    @SzymonRoziewski if perplexity is computed with smoothing then for unknown word the output won't be inf but a larger value. Just use Laplace (from nltk.lm import Laplace) instead of MLE.
    – hafiz031
    Commented Jan 3, 2022 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.