0

I need to scan 2 files in python and say which words in file1 are also in file2. I made a list with all words from file2 and then scan if the line from file1 is in the list.

So this works perfectly, but large files (like 500k) it can take 1h+ and I was wondering if there is a faster way

Thanks in advance

(defined var etc and files)
a = []
for line in var:
    a += [line]
teller = 0

for line1 in new_file:
    if line1 not in a:
        print(line1, file=filter, end='')
    else:
        teller += 1
        print(line1, file=bad, end='' )

print('There were', teller, 'lines that were in the old file.')

3 Answers 3

2

A faster alternative is using sets (as long as you can keep the content of both files in memory):

with open('a.txt', 'r') as a, open('b.txt', 'r') as b:
    a_content = set(a)
    b_content = set(b)

result = a_content.intersection(b_content)
0

If you're worried about speed, then you should be using your OS facilities, not Python loops. Typically, the fastest way to look for individual lines would be to sort both files and then do a simple file diff. If you insist on using Python, that would also be a much quicker way.

0

Your method will work, but it's super inefficient because you are traversing through the file2 for every single word/line within file1. Try turning both file1 and file2 to sets and then compare the sets; I'm pretty sure Python has something like .intersect or .intersection to compare two sets, lists, arrays, or other data structures.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.