6

I'm trying to make a python script that takes a string and gives the count of consecutive words. Let's say:

string = " i have no idea how to write this script. i have an idea."

output = 
['i', 'have'] 2
['have', 'no'] 1
['no', 'idea'] 1
['idea', 'how'] 1
['how', 'to'] 1
['to', 'write'] 1
...

I'm trying to use python without importing collections, counters from collections. What I have is below. I'm trying to use a re.findall(#whatpatterndoiuse, string) to iterate through the string and compare it but I'm having difficulties figuring out how to.

string2 = re.split('\s+', string. lower())
freq_dict = {} #empty dictionary
for word in word_list:
    word = punctuation.sub("", word)
    freq_dic[word] = freq_dic.get(word,0) + 1

freq_list = freq_dic.items()
freq_list.sort()
for word, freq in freq_list:
    print word, freq

Using counter from collections which I did not want. Also it produce an output in a format that is not the one I stated above.

import re
from collections import Counter
words = re.findall('\w+', open('a.txt').read())
print(Counter(zip(words,words[1:])))
7
  • what do you mean by "consecutive words"? Why the lists of two words? Nov 15, 2015 at 18:54
  • 1
    as it would reproduce pairs of consecutive words. such as in the following string "I am what I am" would become (I, am), (am, what), (what, I), (i am)
    – vee
    Nov 15, 2015 at 20:58
  • but why? I guess I'm asking because based on what you posted above, it really just seems like you want the total count of words in the string, which is super easy. Or do you want the frequency that pairs of words appear together? Nov 15, 2015 at 22:37
  • im trying to not use a counter
    – vee
    Nov 15, 2015 at 22:46
  • What I'm asking is what you want to achieve. If all you want is the number of words, and don't want to use a counter, there are way easier ways to go about it. Nov 15, 2015 at 22:50

4 Answers 4

7

Solving this without zip is fairly simple. Just build tuples of each pair of words and track their count in a dict. There are just a few special cases to watch for - when the input string only has one word, and when you are at the end of the string.

Give this a shot:

def freq(input_string):
    freq = {}
    words = input_string.split()
    if len(words) == 1:
        return freq

    for idx, word in enumerate(words):
        if idx+1 < len(words):
            word_pair = (word, words[idx+1])
            if word_pair in freq:
                freq[word_pair] += 1
            else:
                freq[word_pair] = 1

    return freq
3

You need to solve three problems:

  1. generate all pairs of words (['i', 'have'], ['have', 'no'], ...);
  2. count the occurrences of these pair of words;
  3. sort the pairs from the most common to the least common.

The second problem can be easily solved by using a Counter. Counter objects also provide a most_common() method to solve the third problem.

The first problem can be solved in many ways. The most compact way is using zip:

>>> import re
>>> s = 'i have no idea how to write this script. i have an idea.'
>>> words = re.findall('\w+', s)
>>> pairs = zip(words, words[1:])
>>> list(pairs)
[('i', 'have'), ('have', 'no'), ('no', 'idea'), ...]

Putting everything together:

import collections
import re

def count_pairs(s):
    """
    Returns a mapping that links each pair of words
    to its number of occurrences.
    """
    words = re.findall('\w+', s.lower())
    pairs = zip(words, words[1:])
    return collections.Counter(pairs)

def print_freqs(s):
    """
    Prints the number of occurrences of word pairs
    from the most common to the least common.
    """
    cnt = count_pairs(s)
    for pair, count in cnt.most_common():
        print list(pair), count

EDIT: I realized just now that I accidentally read "with collections, counters, ..." instead of "with out importing collections, ...". My bad, sorry.

5
  • thanks for your answer though :) . so sorry for my terrible grammar.
    – vee
    Nov 15, 2015 at 18:27
  • thanks also for your logic flow it really did help me get an idea of where to start. I ran your code also using collections import but it doesnt seem to work am i missing something?
    – vee
    Nov 15, 2015 at 18:37
  • @ValerioZhang: the problem here is that my count_pairs() does not strip punctuation, so that idea and idea. are considered distinct words. Using regular expressions as you were suggesting is an excellent way to solve this problem Nov 15, 2015 at 19:47
  • I found out how to use a counter to implement my function. Only problem is that it doesn't reproduce the format of the output i wanted. I editted it to my original code
    – vee
    Nov 15, 2015 at 20:57
  • @ValerioZhang: I updated the answer. Now it should produce the exact output you want. Nov 15, 2015 at 22:28
0
string = "i have no idea how to write this script. i have an idea."
def count_words(string):
    ''' warning, won't work with a leading or trailing space,
    though all you would have to do is check if there is one, and remove it.'''
    x = string.split(' ')
    return len(x)
1
  • 1
    You can use .strip() to remove leading and trailing spaces.
    – tknickman
    Jan 2, 2017 at 14:43
0

I figured it out answer is posted below. :). It takes a TXT file, but that can easily be manipulated to take in a string. Simple remove arg1 and insert your own string !!!

script, arg1 = argv #takes 2 arguments
#conditions
try:
        sys.argv[1]
except IndexError:
        print('doesnt work insert 2 arguments\n')
        exit()



with open(arg1, 'r') as content_file: #open file
        textsplit = content_file.read() #read it
        textsplit = textsplit.lower() #lowercase it
word_list = textsplit.split() #split file put into var word_lists

textsplit = re.sub(r"[^\w\s]+", "", textsplit).split() #remove white space

#print textsplit

freq_dic = {} #creates empty dictionary

for i in range( 0, len(textsplit)-1): #counter to itterate
        key = textsplit[i] + ',' + textsplit[i+1] # produces corresponding keys
        try:
                freq_dic[key]+=1 #if
        except:
                freq_dic[key]=1 #if not

for word in freq_dic:

        print [word], freq_dic[word]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.