1

I would like to first extract repeating n-grams from within a single sentence using Gensim's Phrases, then use those to get rid of duplicates within sentences. Like so:

Input: "Testing test this test this testing again here testing again here"

Desired output: "Testing test this testing again here"

My code seemed to have worked for generating up to 5-grams using multiple sentences but whenever I pass it a single sentence (even a list full of the same sentence) it doesn't work. If I pass a single sentence, it splits the words into characters. If I pass the list full of the same sentence, it detects nonsense like non-repeating words while not detecting repeating words.

I thought my code was working because I used like 30MB of text and produced very intelligible n-grams up to n=5 that seemed to correspond to what I expected. I have no idea how to tell its precision and recall, though. Here is the full function, which recursively generates all n-grams from 2 to n::

def extract_n_grams(documents, maximum_number_of_words_per_group=2, threshold=10, minimum_count=6, should_print=False, should_use_keywords=False):
    from gensim.models import Phrases
    from gensim.models.phrases import Phraser

    tokens = [doc.split(" ") for doc in documents] if type(documents) == list else [documents.split(" ") for _ in range(100)] # this is what I tried

    final_n_grams = []
    for current_n in range(maximum_number_of_words_per_group - 1):
        n_gram = Phrases(tokens, min_count=minimum_count, threshold=threshold, connector_words=connecting_words)

        n_gram_phraser = Phraser(n_gram)

        resulting_tokens = []
        for token in tokens:
            resulting_tokens.append(n_gram_phraser[token])

        current_n_gram_final = []
        for token in resulting_tokens:
            for word in token:
                if '_' in word:
                    # no n_gram should have a comma between words
                    if ',' not in word:
                        word = word.replace('_', ' ')

                        if word not in current_n_gram_final and all([word not in gram for gram in final_n_grams]):
                            current_n_gram_final.append(word)

        tokens = n_gram[tokens]

        final_n_grams.append(current_n_gram_final)

In addition to trying repeating the sentence in the list, I also tried using NLKT's word_tokenize as suggested here. What am I doing wrong? Is there an easier approach?

1 Answer 1

1

The Gensim Phrases class is designed to statistically detect when certain pairs of words appear so often together, compared to independently, that it might be useful to combine them into a single token.

As such, it's unlikely to be helpful for your example task, of eliminating the duplicate 3-word ['testing', 'again', 'here'] run-of-tokens.

First, it never eliminates tokens – only combines them. So, if it saw the couplet ['again', 'here'] appearing ver often together, rather than as separate 'again' and 'here', it'd turn it into 'again_here' – not eliminate it.

But second, it does these combinations not for every repeated n-token grouping, but only if the large amount of training data implies, based on the threshold configured, that certain pairs stick out. (And it only goes beyond pairs if run repeatedly.) Your example 3-word grouping, ['testing', 'again', 'here'], does not seem likely to stick out as a composition of extra-likely pairings.

If you have a more rigorous definition of which tokens/runs-of-tokens need to be eliminated, you'd probably want to run other Python code on the lists-of-tokens to enforce that de-duplication. Can you describe in more detail, perhaps with more examples, the kinds of n-grams you want removed? (Will they only be at the beginning or end of a text, or also the middle? Do they have to be next-to each other, or can they be spread throughout the text? Why are such duplicates present in the data, & why is it thought important to remove them?)

Update: Based on the comments about the real goal, a few lines of Python that check, at each position in a token-list, whether the next N tokens match the previous N tokens (and thus can be ignored) should do the trick. For example:

def elide_repeated_ngrams(list_of_tokens):
    return_tokens = [] 
    i = 0
    while i < len(list_of_tokens):
        for candidate_len in range(len(return_tokens)):
            if list_of_tokens[i:i+candidate_len] == return_tokens[-candidate_len:]:
                i = i + candidate_len  # skip the repeat
                break  # begin fresh forward repeat-check
        else:
            # this token not part of any repeat; include & proceed
            return_tokens.append(list_of_tokens[i])
            i += 1
    return return_tokens 

On your test case:

>>> elide_repeated_ngrams("Testing test this test this testing again here testing again here".split())
['Testing', 'test', 'this', 'testing', 'again', 'here']
8
  • Thank you very much for replying! I am aware that it does not remove repetitions. I intended to use the resulting n-grams to remove them myself. I get it but why does it not work when I take a single sentence and feed only it to Gensim n times? Shouldn't it detect that the repeated n-grams appear a bunch of times because they are present in every sentence and and then classify it as n_gram, which would later allow me to use them to get rid of repetitions? I want literally EVERY word or multiple-word repetition, no matter the type, to be reduced to a single occurrence. Commented Oct 30, 2021 at 21:19
  • Well, it's not designed for reporting all n-grams - just surveying them, then performing a statistical-based replacement. So in some ways it's overkill for a desired endpoint of removing "every word or multiple-word repetition", but others it lacks steps that process might need. It definitely won't do anything interesting from repeating the same sentence N times - because that leaves relative unigram vs pairs frequencies the same no matter what N is, and all of its intended benefits depend on the kinds of relative, smooth, varied frequencies in large, diverse natural-langauge text.
    – gojomo
    Commented Oct 30, 2021 at 22:26
  • What's more, "EVERY word or multiple-word repetition, no matter the type, to be reduced to a single occurrence" is still under-specified. Do you want the sentence "roses are red violets are blue" to remove the 2nd 'are'? For "roses are red, roses are thorny" should the 2nd 'roses are' be removed? Etc. And, why are such repetitions even an issue? (Often, text representations want to overweight words that appear more often.)
    – gojomo
    Commented Oct 30, 2021 at 22:29
  • 1
    Yes, only adjacent. I had thought about such cases and ALL repetitions should count as long as they are adjacent. I was hoping there was something already made for this but, of course, as you said yourself, it's quite simple and I will develop it myself in a day or so once I get to this part. Thank you so much for all your help, I really appreciate it! Commented Oct 31, 2021 at 23:55
  • 1
    Thank you very much! That is really kind of you :D Commented Nov 1, 2021 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.