0

I have a database of words and a dataset with text lines. Every time there is a word in the line of the text file that appears in the words file as well, I want to do a trick. My code looks like:

import re
f = open(r"words.txt")
print len(flist)
d = open(r"text.txt", "r")
dlist = d.readlines()

for line in flist:
    lowline = line.lower()
    for word in dlist:
        lowword = word.lower()
        if lowword in lowline:
            *trick*

However, this code finds no matches, altough there are many words that are exactly the same. Any thoughts on this one?

4
  • Have you mixed up your files and variables? The word variable appears to be read from the text.txt file, whereas line is from words.txt, which seems to suggest that you need to swap them around. Jun 26, 2013 at 18:41
  • 2
    readlines returns the lines with a newline at the end of the string. you're not going to find 'cat\n' inside 'my cat is black\n'
    – jterrace
    Jun 26, 2013 at 18:41
  • @andersschuller Sorry, I messed it up in this post because I wanted to simplify the file names. Jun 26, 2013 at 18:48
  • @jterrace Yeah, that was the problem. Thanks! Jun 26, 2013 at 18:48

1 Answer 1

0

Save the words from database into a set first and apply str.strip and str.lower to them. str.strip will remove the leading and trailing white space characters like '\n',etc.

Sets provide O(1) lookup, and also set intersections is going to be much more efficient than your current O(n^2) approach.

Then iterate over each line in words file and apply str.strip and str.lower first before searching for it in the set.

with open(r"words.txt") as f1, open(r"text.txt", "r") as f2:

    dlist = set(line.strip().lower() for line in f2)  #set of words from database
    for line in f1:
        line = line.strip().lower()     #use strip to remove '\n'
        words = set(line.split())    #use split to get the words from the line
                                     #and convert it into a set
        common_words = words & dlist  #use set intersection to find common words
        for word in common_words:  
           *trick* 

Please replace f1 and f2 appropriately as I am confused which one is database and which one is text dataset.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.