2

I have several large text text files that all have the same structure and I want to delete the first 3 lines and then remove illegal characters from the 4th line. I don't want to have to read the entire dataset and then modify as each file is over 100MB with over 4 million records.

Range   150.0dB -64.9dBm
Mobile unit 1   Base    -17.19968    145.40369  999.8
Fixed unit  2   Mobile  -17.20180    145.29514  533.0
Latitude    Longitude   Rx(dB)  Best unit
-17.06694    145.23158  -050.5  2
-17.06695    145.23297  -044.1  2

So lines 1,2 and 3 should be deleted and in line 4, "Rx(db)" should be just "Rx" and "Best Unit" be changed to "Best_Unit". Then I can use my other scripts to geocode the data.

I can't use commandline programs like grep (as in this question) as the first 3 lines are not all the same -the numbers (such as 150.0dB, -64*) will change in each file so you have to just delete the whole of lines 1-3 and then grep or similar can do the search-replace on line 4.

Thanks guys,

=== EDIT new pythonic way to handle larger files from @heltonbiker. Error.

import os, re
##infile = arcpy.GetParameter(0)
##chunk_size = arcpy.GetParameter(1) # number of records in each dataset

infile='trc_emerald.txt'
fc= open(infile)
Name = infile[:infile.rfind('.')]
outfile = Name+'_db.txt'

line4 = fc.readlines(100)[3]
line4 = re.sub('\([^\)].*?\)', '', line4)
line4 = re.sub('Best(\s.*?)', 'Best_', line4)
newfilestring = ''.join(line4 + [line for line in fc.readlines[4:]])
fc.close()
newfile = open(outfile, 'w')
newfile.write(newfilestring)
newfile.close()

del lines
del outfile
del Name
#return chunk_size, fl
#arcpy.SetParameterAsText(2, fl)
print "Completed"

Traceback (most recent call last): File "P:\2012\Job_044_DM_Radio_Propogation\Working\FinalPropogation\TRC_Emerald\working\clean_file_1c.py", line 13, in newfilestring = ''.join(line4 + [line for line in fc.readlines[4:]]) TypeError: 'builtin_function_or_method' object is unsubscriptable

4
  • 3
    Python is overkill. Use sed.
    – wim
    Feb 27, 2012 at 23:18
  • Why is Python a requirement here. This seems like a great job for UNIX commands. Not sure what environment you're on, but yeah.
    – Brian
    Feb 27, 2012 at 23:19
  • I am using arcpy in the follow up steps to geocode/buffer etc the dataset (see gis.stackexchange.com/questions/20892/… for details). The issue with SED and GREP is that I don't see a way of deleting the first three non-unique lines -a simple search and replace won't work as per the last paragraph of my question (just added).
    – GeorgeC
    Feb 27, 2012 at 23:27
  • Unless you edit in place all the characters from first 4 rows with spaces (not exactly what you asked for but might for what you need) or rewrite all the subsequent lines 4 lines up, truncate and resave the file (bad idea) your best bet is to go with Gordon's solution with sed. Feb 27, 2012 at 23:57

3 Answers 3

9

As wim said in the comments, sed is the right tool for this. The following command should do what you want:

sed -i -e '4 s/(dB)//' -e '4 s/Best Unit/Best_Unit/' -e '1,3 d' yourfile.whatever

To explain the command a little:

-i executes the command in place, that is it writes the output back into the input file

-e execute a command

'4 s/(dB)//' on line 4, subsitute '' for '(dB)'

'4 s/Best Unit/Best_Unit/' same as above, except different find and replace strings

'1,3 d' from line 1 to line 3 (inclusive) delete the entire line

sed is a really powerful tool, which can do much more than just this, well worth learning.

14
  • I'm not sure that will avoid reading the whole file as asked by the OP.
    – jcollado
    Feb 27, 2012 at 23:33
  • 1
    That's a good point. It would be interesting to know how sed handles that kind of thing, although I hope it would be smart and ignore all lines other than 1-4. Feb 27, 2012 at 23:40
  • 1
    To some extent, you can't not read the entire file. How can you match something without reading it? Sed doesn't read the whole file into memory, so I think your answer is a good one.
    – prelic
    Feb 28, 2012 at 0:38
  • I tried it and get an error which I can't figure out - C:\Radio Mobile\Output\Ian_Test>sed -i -e '4 s/(dB)//' it/' -e '1,3 d' trc_Longlands_10m.txt sed: -e expression #1, char 1: unknown command: `'' --- I tried it in individual components but it still gives same error.
    – GeorgeC
    Feb 28, 2012 at 2:02
  • this is a total guess, but maybe try replacing '4 s/(dB)//' with '4,4 s/(dB)//' Feb 28, 2012 at 2:07
1

You can use file.readlines() with an aditional argument in order to read just a few first lines from the file. From the docs:

f.readlines() returns a list containing all the lines of data in the file. If given an optional parameter sizehint, it reads that many bytes from the file and enough more to complete a line, and returns the lines from that. This is often used to allow efficient reading of a large file by lines, but without having to load the entire file in memory. Only complete lines will be returned.

Then the most robust way to manipulate generic strings are Regular Expressions. In Python, this means the re module with, for example, the re.sub() function.

My suggestion, which should be adapted to suit your needs:

import re

f = open('somefile.txt')
line4 = f.readlines(100)[3]
line4 = re.sub('\([^\)].*?\)', '', line4)
line4 = re.sub('Best(\s.*?)', 'Best_', line4)
newfilestring = ''.join(line4 + [line for line in f.readlines[4:]])
f.close()
newfile = open('someotherfile.txt', 'w')
newfile.write(newfilestring)
newfile.close()
4
  • thanks...I am trying this solution as the pythonic way from cjrh doesn't work on my new larger files. I am getting an error (below), your adapted code is in edit to question. Any ideas?
    – GeorgeC
    Mar 11, 2012 at 22:53
  • @GeorgeC where it says [line for line in fc.readlines[4:]] it should be [line for line in fc.readlines()[4:]]. (note the parentheses after readline - it is a function, must be used like readline(), and then its result indexed, like readline()[4:] Mar 12, 2012 at 12:03
  • Also, it might be faster, if you figure it out how, to use f.read() instead of [l for l in f.readlines()]. The problem is that you need to do f.seek(p) before, where p is the position in the file where the 4th line begins. Mar 12, 2012 at 12:04
  • just tried on a 1gb file -still get a memory error. Any other ideas - SED works fine on the same file but isn't as easy to use.
    – GeorgeC
    Mar 15, 2012 at 3:47
1

Just try it for each file. 100 MB per file is not that big, and as you can see, the code to just make an attempt is not time-consuming to write.

with open('file.txt') as f:
  lines = f.readlines()
lines[:] = lines[3:]
lines[0] = lines[0].replace('Rx(db)', 'Rx')
lines[0] = lines[0].replace('Best Unit', 'Best_Unit')
with open('output.txt', 'w') as f:
  f.write('\n'.join(lines))
6
  • 1
    First of all, with such a big file I would not read the whole file at once this way but rather use line reading in a generator loop. This way you don't have to use up 100 MB of RAM (plus the baggage PyObject and memory fragmentation add to it).
    – Frg
    Feb 28, 2012 at 0:03
  • @Frg: how will you save the modifications? Feb 28, 2012 at 0:44
  • It's not about not reading the rest of the file, but doing so on a line-by-line basis. My comment would probably make more sense if the file was more like 1 GB in size.
    – Frg
    Feb 28, 2012 at 17:13
  • @cjrh -thank for this. It works great. How can I split the dataset into 1 million line blocks at the same time? arcgis is caughing at trying to buffer and process my files which are about 500mb with 10million lines. see stackoverflow.com/questions/9612882/… . BTW your python solution takes about the same time as cygwin (~3 mins) to run.
    – GeorgeC
    Mar 8, 2012 at 4:36
  • My base txt databases grew to over 1gb now with 30m lines and the python option fails due to memory issues while the sed solution works with no issues. Wish I could select both your and @Gordon Bailey answer as I use both. Can you help fix the python?
    – GeorgeC
    Mar 11, 2012 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.