0

I have a raw binary file that is several gigs and I'm attempting to process it in chunks. Before I can start processing the data I have to remove the header that it has. None of the string methods like .find or checking for string in data chunk works because of the raw binary file format. I would like to automatically strip the header but it can vary in length and my current approach of looking for the last new line character didn't work because the raw binary data has matching bits in the data.

Data format:
BEGIN_HEADER\r\n
header of various line count\r\n
HEADER_END\r\n raw data starts here

how i'm reading in the file

filename="binary_filename"
chunksize=1024
with open(filename, "rb") as f:
    chunk = f.read(chunksize)
    for index, byte in enumerate(chunk):
        if byte == ord('\n'):
            print("found one " + str(index))

Is there a simple way to extract the HEADER_END\r\n line without sliding a byte array through the file? current approach:

chunk = f.read(chunksize)
index=0
not_found=True
while not_found:
    if chunk[index:index+12] == b'HEADER_END\r\n':
        print("found")
        not_found=False
    index+=1

1 Answer 1

1

You could use linecache:

import linecache
currentline = 0
while(linecache.getline("file.bin",currentline)!="HEADER_END\n"):
    currentline=currentline+1

#print raw data
currentline = currentline + 1
rawdata = linecache.getline("file.bin",currentline)
currentrawdata = rawdata
while(currentrawdata):
    currentrawdata = linecache.getline("file.bin",currentline+1)
    rawdata = rawdata + currentrawdata
    currentline = currentline + 1
print rawdata

UPDATE

We can split the problem in two, first we can remove the header, then we can read it into chunks:

lines= open('test_file.bin').readlines()
currentline = 0
while(lines[currentline] != "HEADER_END\r\n"):
     currentline=currentline+1
open('newfile.bin', 'w').writelines(lines[currentline:-1])

A file ( newfile.bin ) will be created containing just the raw data. Now it can be read direclty in chunks:

chunksize=1024
with open('newfile.bin', "rb") as f:
    chunk = f.read(chunksize)

UPDATE 2

It is also possible to do this without using the intermediate file:

#defines the size of the chunks
chunksize=20
filename= 'test_file.bin'
endHeaderTag = "HEADER_END\r\n"
#Identifies at which line there is HEADER_END
lines= open(filename).readlines()
currentline = 0
while(lines[currentline] != endHeaderTag):
     currentline=currentline+1
currentline=currentline+1
#Now currentline contains the index of the first line to the raw data

#With the reduce operation we generate a single string from the list of lines
#we are considering only the lines after the currentline
header_stripped = reduce(lambda x,y:x+y,lines[currentline:])

#Lastly we read successive chunks and we store them into the chunk list.
chunks = []
reminder = len(header_stripped)%chunksize
for i in range(1,len(header_stripped)/chunksize + reminder):
    chunks.append( header_stripped[(i-1)*chunksize:i*chunksize])
4
  • there are no new line tags for the remainder of the large binary file. Will it attempt to read the remainder of the file as a single line? Jun 28, 2016 at 16:48
  • I didn't get that there could be new lines in the rawdata part, I updated the answer.
    – Stram
    Jun 28, 2016 at 16:58
  • in the event that there are no new lines in the data will the rawdata be read in as one massive chunk? In this case that chunk would be several gigs. Jun 28, 2016 at 17:11
  • With what I wrote here yes, it will consider everything as a single line and will read all the rawdata after the header at once.
    – Stram
    Jun 28, 2016 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.