1

I have a program which has the purpose of reading eight files that are a million characters long, no punctuation, just a bunch of characters.

The eight file represent four DNA samples found, and what the program does is takes the characters from one file in a sample, and combines them with the characters in the other file of the same sample. So for example, if file1 read:

abcdefg

and file2 read:

hijklmn

the combination would be:

ah, bi, cj, dk, el, fm, gn

At any rate, the program then goes on to count how many combinations of each pair exists, and it will print out a dictionary that would read something like this for example:

{'mm': 52, 'CC': 66, 'SS': 24, 'cc': 19, 'MM': 26, 'ss': 58, 'TT': 43, 'tt': 32}

The problem is, while the program works fine for small files, for the large million character long (yes, that is a literal number, not hyperbole) files, the program hangs, and doesn't seems to ever get to finishing the task. (I left it running overnight once and nothing came of it.)

Is it an overflow error, or the method I'm using is too small for a large file? Is there a better way to handle this?

My code:

import re
from collections import Counter

def ListStore(fileName):
    '''Purpose, stores the contents of file into a single string'''           

    #old code left in for now
    '''
    with open(fileName, "r") as fin:
        fileContents = fin.read().rstrip()
        fileContents = re.sub(r'\W', '', fin.read())
    '''
    #opens up the file given to the function
    fin = open(fileName,'r')

    #reads the file into a string, strips out the newlines as well
    fileContents = fin.read().rstrip()


    #closes up the file
    fin.close()

    #splits up the fileContents into a list of characters
    fileContentsList = list(fileContents)   

    #returns the string
    return fileContentsList


def ListCombo(list1, list2):
    '''Purpose: combines the two DNA lists into one'''


    #creates an empty dictionary for list3
    list3 = []

    #combines the codes from one hlaf with their matching from the other
    list3 = [''.join(pair) for pair in zip(list1, list2)]

    return list3


def printResult(list):
    '''stores the result of the combination in a dictionary'''




    #stores the result into a dictionary
    result = dict((i,list.count(i)) for i in list)

    print (result)
    return result


def main():

    '''Purpose: Reads the contents of 8 files, and finds out how many
    combinations exist'''


    #first sample files

    file_name = "a.txt"
    file_name2 = "b.txt"

    #second sample files
    file_name3 = "c.txt"
    file_name4 = "d.txt"

    #third sample files
    file_name5 = "e.txt"
    file_name6 = "f.txt"

    #fourth sample files
    file_name7 = "g.txt"
    file_name8 = "h.txt"


    #Get the first sample ready

    #store both sides into a list of characters

    contentList = ListStore(file_name)

    contentList2 = ListStore(file_name2)

    #combine the two lists together
    combo_list = ListCombo(contentList, contentList2)

    #store the first sample results into a dictionary
    SampleA = printResult(combo_list)

    print (SampleA)

    # ****Get the second sample ready****

    #store both sides into a list of characters
    contentList3 = ListStore(file_name3)
    contentList4 = ListStore(file_name4)

    #combine the two lists together
    combo_list2 = ListCombo(contentList3, contentList4)

    #store the first sample results into a dictionary
    SampleB = printResult(combo_list2)

    print (SampleB)

    # ****Get the third sample ready****

    #store both sides into a list of characters
    contentList5 = ListStore(file_name5)
    contentList6 = ListStore(file_name6)

    #combine the two lists together
    combo_list3 = ListCombo(contentList5, contentList6)

    #store the third sample results into a dictionary
    SampleC = printResult(combo_list3)

    print (SampleC)

    # ****Get the second sample ready****

    #store both sides into a list of characters
    contentList7 = ListStore(file_name7)
    contentList8 = ListStore(file_name8)

    #combine the two lists together
    combo_list4 = ListCombo(contentList7, contentList8)

    #store the fourth sample results into a dictionary
    SampleD = printResult(combo_list4)

    print (SampleD)



if __name__ == '__main__':
    main()

3 Answers 3

2

Don't read the whole contents into memory. There is no need. Moreover, zip() will already split your strings into characters, so you don't need to do this yourself.

The trick here is to create a generator that pairs off your characters while reading the two files in chunks, which would be the most efficient way to read the files.

Lastly, use collections.Counter() to keep counts:

from functools import partial
from collections import Counter

with open(filename1, 'r') as file1, open(filename2, 'r') as file2:
    chunked1 = iter(partial(file1.read, 1024), '')
    chunked2 = iter(partial(file2.read, 1024), '')
    counts = Counter(''.join(pair) for chunks in zip(chunked1, chunked2) for pair in zip(*chunks))

Here the code is read in chunks of 1024 bytes; adjust as needed for best performance. No more than 2048 bytes from the files are held in memory at once, pairs are generated on the fly as they are counted.

1

In your printResult method, you go though each element i in list, and assign the value list.count(i) to the key i in your result dictionary.

I'm not entirely sure how count(i) works, but I believe it involves searching through most of the list, and counting the number of elements i EACH TIME IT RUNS. In your code, if you have a duplicate, like in ['aa','bb','aa'], you will count how many elements 'aa' there are in the list twice, going though the entire list each time. This is VERY time consuming in long lists.

You only need to go through the list once in order to count how many elements of each type there are. I would suggest using a defaultdict for this, because you can make each new key start with the default value 0.

    from collections import defaultdict
    result = defaultdict(int)
    for i in list:
        result[i] = result[i] + 1
    print result

Creating a defaultdict with int allows each new key to start out with the value 0. You can then traverse the list once, adding 1 to the value for each pair each time you find it. This eliminates going through the list more than once.

1

As written, I personally don't think your program is I/O bound -- and even if it was, breaking that up into many calls, even if buffered, wouldn't be as fast as reading the whole thing into memory as you were doing. That said, I'm not exactly sure why your program takes so long to process the huge files -- it could be the many unneeded operations it's doing because strings and lists are both sequences, so there's often no need to convert from one to another.

Here's an optimized version of your program with most of the redundant and/or unnecessary stuff removed. It actually utilizes thecollections.Counterclass imported in your code but is never used, and, even though it still reads the contents of files into memory, it only retains these for the minimum amount of time required to process each pair of them.

from collections import Counter
import os

DATA_FOLDER = 'datafiles' # folder path to data files ('' for current dir)

def ListStore(fileName):
    '''return contents of file as a single string with any newlines removed'''
    with open(os.path.join(DATA_FOLDER, fileName), 'r') as fin:
        return fin.read().replace('\n', '')

def ListCombo(seq1, seq2):
    '''combine the two DNA sequences into one'''
    # combines the codes from one half with their matching from the other
    return [''.join(pair) for pair in zip(seq1, seq2)]

def CountPairs(seq):
    '''counts occurences of pairs in the list of the combinations and stores
    them in a Counter dict instance keyed by letter-pairs'''
    return Counter(seq)

def PrintPairs(counter):
    #print the results in the counter dictionary (in sorted order)
    print('{' + ', '.join(('{}: {}'.format(pair, count)
        for pair, count in sorted(counter.items()))) + '}')

def ProcessSamples(file_name1, file_name2):
    # store both sides into a list of characters
    contentList1 = ListStore(file_name1)
    contentList2 = ListStore(file_name2)

    # combine the two lists together
    combo_list = ListCombo(contentList1, contentList2)

    # count the sample results and store into a dictionary
    counter = CountPairs(combo_list)

    #print the results
    PrintPairs(counter)

def main():
    '''reads the contents of N pairs of files, and finds out how many
    combinations exist in each'''
    file_names = ('a.txt', 'b.txt',
                  'c.txt', 'd.txt',
                  'e.txt', 'f.txt',
                  'g.txt', 'h.txt',)

    for (file_name1, file_name2) in zip(*([iter(file_names)]*2)):
        ProcessSamples(file_name1, file_name2)

if __name__ == '__main__':
    main()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.