9

I have 2 codes which did the same work as which i am asking , but still i didn't get any useful or better code for my data set to make it useful for me , First let me clear what i am doing . I have 2 TEXT files , one name as input_num and second named as input_data as it is clear from names that input_num.txt have number in them , and input_data have data in it , these 2 files are of 8 to 10 mb , let me show you some of their part , This is 'input_num.txt'

ASA5.txt DF4E6.txt DFS6Q7.txt

and this input_data.txt

>56|61|83|92|ASA5
Dogsarebarking

These 2 are some parts of their text files , input_data.txt have last column which contain ASA5 and so on , these are data from input_num.txt , so the program first check the last column of >56|61|83|92|ASA5 which is ASA5 than goto input_num.txt which have 5 , it contain some value in input_num.txt like 4 in the above , so it come back to the input_data.txt goto the words and cut them to 4 ,

I have 2 codes for it : 1 is

import os
import re
file_c = open('num_data.txt')
file_c = file_c.read()
lines = re.findall(r'\w+\.txt \d+', file_c)
numbers = {}

for line in lines:
    line_split = line.split('.txt ')
    hash_name = line_split[0]
    count = line_split[1]
    numbers[hash_name] = count
file_i = open('input_data.txt')
file_i = file_i.read()

for hash_name, count in numbers.iteritems():
    regex = '(' + hash_name.strip() + ')'
    result = re.findall(r'>.*\|(' + regex + ')(.*?)>', file_i, re.S)

    if len(result) > 0:
        data_original = result[0][2]
        stripped_data = result[0][2][int(count):]
        file_i = file_i.replace(data_original, '\n' + stripped_data)
f = open('input_new.txt', 'wt')
f.write(file_i)
f.close()

and the 2nd is

import csv
output = open('output.txt' , 'wb')
def get_min(num):
    return int(open('%s.txt' % num, 'r+').readlines()[0])
last_line = ''
input_list = []

#iterate over input.txt in sort the input in a list of tuples 
for i, line in enumerate(open('input.txt', 'r+').readlines()): 
    if i%2 == 0: 
        last_line = line
    else:
        input_list.append((last_line, line))
filtered = [(header, data[:get_min(header[-2])] + '\n' ) for (header, data) in input_list]
[output.write(''.join(data)) for data in filtered]
output.close()
10
  • 4
    What's your question/problem here? You say you have two working programs, and "Both programs work better but not accurate for me , mean to say not on huge data". So… what do they do wrong? What's the expected vs. observed output? Can you identify what about the output causes a problem? (And ideally give us a stripped-down sample input that reproduces it; if not, can you post the 10MB files somewhere in case someone wants to debug your code?)
    – abarnert
    Apr 12, 2013 at 18:53
  • 1
    Also, while you say you've chosen to use csv because you feel comfortable with it… you're not actually using csv at all. You do import it in one case, but you never touch it. You use a regex in one program, and apparently use complete lines without even attempting to split into columns in the other.
    – abarnert
    Apr 12, 2013 at 18:55
  • 3
    Finally, if the problem you're having with huge files is either "it's way too slow" or "it takes way too much memory", I'd give 1:1 odds that the problem is your readlines() calls. There is almost never a reason to use readlines(). If you just do for i, line in enumerate(f), it gives you one line at a time, buffering as efficiently as possible; if you do for i, line in enumerate(f.readlines()), it first reads and parses the entire file in memory, and only then gives you one line at a time. This is never helpful, and often a problem.
    – abarnert
    Apr 12, 2013 at 18:56
  • @abarnert In one program in work when i split my input_num.txt into number of text files , and than it again stop working when some text file which is not numeric name appear , like EOG6003RW
    – Rocket
    Apr 12, 2013 at 18:58
  • 3
    No, don't mail it. That won't help anyone else who wants to test your code, or any future readers who think they might have similar problems. Also, email is a bad way for sharing files more than a couple megabytes. Find somewhere you can post them online, and put a URL in the question.
    – abarnert
    Apr 12, 2013 at 19:16

1 Answer 1

5
+50

As far as I could understand from the description of your problem with the first code, you want the first N letters in the output while in fact you get everything except the first N letters. This can probably be fixed by changing

stripped_data = result[0][2][int(count):]

to

stripped_data = result[0][2][:int(count)]

I also think the regular expressions used are not completely accurate. I suggest the following for the numbers:

with open('num.txt') as nums:
    lines = re.findall(r'\w+\.txt\s+\d+', nums.read())

numbers = {}
for line in lines:
    line_split = re.split(r'\.txt\s+', line)
    count = line_split[1]
    numbers[line_split[0]] = int(line_split[1])

and the following for the data:

with open('input_data.txt') as file_i:
     data = file_i.read()

for name, count in numbers.iteritems():
    result = re.search(r'\|{}\n(.*?)(>|$)'.format(name), s, re.S)
    if result:
        data_original = result.group(1)
        stripped_data = data_original[:count]
        data = data.replace(data_original, stripped_data)
with open('input_new.txt', 'w') as f:
    f.write(data)

But note that the idea is still flawed because you can accidentally change more than one sequence when doing replace. Also this method is memory-inefficient because the files are read into the memory as one string. I suggest to use an iterative parser for the data, like the ones I mention below.


Anyway, if I had to solve this problem, I'd use pyteomics to read and write FASTA files (because I wrote it and always have it handy).

The format of input_num.txt is awful, so I think the code from your first example is the best one can do to extract the info. I made some fixes to it though:

import re
from pyteomics import fasta

with open('num.txt') as nums:
    lines = re.findall(r'\w+\.txt\s+\d+', nums.read())

numbers = {}
for line in lines:
    line_split = re.split(r'\.txt\s+', line)
    count = line_split[1]
    numbers[line_split[0]] = int(line_split[1])

with fasta.read('data.txt') as data:
    new_data = ((header, seq[:numbers.get(header.rsplit('|', 1)[-1])])
            for header, seq in data)
    fasta.write(new_data, 'new_data.txt')

On the other hand, since your data look more like DNA sequences and pyteomics is for proteomics, it may make more sense to use BioPython.SeqIO:

import re
from Bio import SeqIO

with open('num.txt') as nums:
    lines = re.findall(r'\w+\.txt\s+\d+', nums.read())

numbers = {}
for line in lines:
    line_split = re.split(r'\.txt\s+', line)
    count = line_split[1]
    numbers[line_split[0]] = int(line_split[1])
data = SeqIO.parse(open('data.txt'), 'fasta')

def new_records():
    for record in data:
        record.seq = record.seq[:numbers.get(record.description.rsplit('|', 1)[-1])]
        yield record

with open('new_data.txt', 'w') as new_data:
    SeqIO.write(new_records(), new_data, 'fasta')
3
  • When i downloaded the exe of pyteomics for python 2.7 , it install , and when i import it from shell it import without error but when i import it in script it show me the error of no module name fasta
    – Rocket
    Apr 17, 2013 at 10:08
  • @Angel Does from pyteomics import fasta work in shell? (Also I just expanded the answer in the part about your code) Apr 17, 2013 at 10:28
  • @Angel then it should work in a script, too, if you run it with the same Python installation that is used for the shell. Apr 17, 2013 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.