I have written a file processing program and now it needs to read from a zipped file(.gz unzipped file may get as large as 2TB),
Is there a sed equivalent for zipped files like (zcat/cat) or else what would be the best approach to do the following efficiently
ONE=`zcat filename.gz| sed -n $counts`
$counts : counter to read(line by line)
The above method works, but is quite slow for large file as I need to read each line and perform the matching on certain fields.
Thanks
EDIT
Though not directly helpful, here are a set of zcommands
http://www.cyberciti.biz/tips/decompress-and-expand-text-files.html
sed
code is completely optimized OR more likely using one of awk,perl,python to do your calculation. Good luck.logrotate
as a method to create date-time-seriesNum stamped filenames. (There are other approaches besides logrotate.) Then you can reduce what you are searching OR at least run parallel processes on multiple files at one time. You'll get better help if you reedit your question as a 2TB processing problem. Good luck!i need to read each line and perform the matching on certain fields
What exactly is the problem? I sound sthat you have a higher level logic which is depending on this to be fast? However have you considered that it could be the higher level logic which could be optimized, and you may want to expand the question to include that?