6

I am attempting to read a binary file using Python. Someone else has read in the data with R using the following code:

x <- readBin(webpage, numeric(), n=6e8, size = 4, endian = "little")
      myPoints <- data.frame("tmax" = x[1:(length(x)/4)],
                             "nmax" = x[(length(x)/4 + 1):(2*(length(x)/4))],
                             "tmin" = x[(2*length(x)/4 + 1):(3*(length(x)/4))],
                             "nmin" = x[(3*length(x)/4 + 1):(length(x))])

With Python, I am trying the following code:

import struct

with open('file','rb') as f:
    val = f.read(16)
    while val != '':
        print(struct.unpack('4f', val))
        val = f.read(16) 

I am coming to slightly different results. For example, the first row in R returns 4 columns as -999.9, 0, -999.0, 0. Python returns -999.0 for all four columns (images below).

Python output: enter image description here

R output: enter image description here

I know that they are slicing by the length of the file with some of the [] code, but I do not know how exactly to do this in Python, nor do I understand quite why they do this. Basically, I want to recreate what R is doing in Python.

I can provide more of either code base if needed. I did not want to overwhelm with code that was not necessary.

3
  • 2
    Your Python code is reading values in groups of 4, but your R code is dividing the values into 4 groups. Aug 22, 2018 at 21:50
  • Per R Documentation, the argument size is: "The number of bytes per element in the byte stream". Does that not mean reading in groups of 4? I learned all of this stuff... today... so I apologize if it is obvious. Thanks for the response.
    – Mark P.
    Aug 22, 2018 at 22:04
  • 3
    No, that means reading 4-byte floats. The 4 groups come from the 4 slicing expressions that each take a quarter of the vector. Aug 22, 2018 at 22:05

2 Answers 2

1

Deducing from the R code, the binary file first contains a certain number tmax's, then the same number of nmax's, then tmin's and nmin's. What the code does is reading the entire file, which is then chopped up in the 4 parts (tmax's, nmax's, etc..) using slicing.

To do the same in python:

import struct

# Read entire file into memory first. This is done so we can count
# number of bytes before parsing the bytes. It is not a very memory
# efficient way, but it's the easiest. The R-code as posted wastes even
# more memory: it always takes 6e8 * 4 bytes (~ 2.2Gb) of memory no
# matter how small the file may be.
#
data = open('data.bin','rb').read()

# Calculate number of points in the file.  This is 
# file-size / 16, because there are 4 numeric()'s per
# point, and they are 4 bytes each.
#
num = int(len(data) / 16)

# Now we know how much there are, we take all tmax numbers first, then
# all nmax's, tmin's and lastly all nmin's.

# First generate a format string because it depends on the number points
# there are in the file. It will look like: "fffff"
#
format_string = 'f' * num

# Then, for cleaner code, calculate chunk size of the bytes we need to 
# slice off each time.
#
n = num * 4     # 4-byte floats

# Note that python has different interpretation of slicing indices
# than R, so no "+1" is needed here as it is in the R code.
#
tmax = struct.unpack(format_string, data[:n])
nmax = struct.unpack(format_string, data[n:2*n])
tmin = struct.unpack(format_string, data[2*n:3*n])
nmin = struct.unpack(format_string, data[3*n:])

print("tmax", tmax)
print("nmax", nmax)
print("tmin", tmin)
print("nmin", nmin)

If the goal is to have this data structured as a list of points(?) like (tmax,nmax,tmin,nmin), then append this to the code:

print()
print("Points:")

# Combine ("zip") all 4 lists into a list of (tmax,nmax,tmin,nmin) points.
# Python has a function to do this at once: zip()
#
i = 0
for point in zip(tmax, nmax, tmin, nmin):
    print(i, ":", point)
    i += 1
4
  • I believe this is working for most, it does not seem to be working for tmin. No errors, but the decoded data makes no sense (all undefined -999). I will keep looking but you did help. If I solve it (or you have ideas), Ill still mark this for the bounty. Thanks for the response.
    – Mark P.
    Aug 28, 2018 at 17:25
  • It would help to have the binary input file (or a way to create one), and the expected results. The R-code does give the correct expected output?
    – Hkoof
    Aug 28, 2018 at 22:13
  • 1
    Thanks. You're welcome. It's fun to try solve it. In the meantime, when I have time, I will try to make a version that does not try to read and copy it entirely into memory. Those files are to big for that. Hope that also helps to sole te problem.
    – Hkoof
    Aug 29, 2018 at 21:04
  • Actually I have a really large box to play with so memory is not an issue... but yes thank you.
    – Mark P.
    Aug 30, 2018 at 12:18
0
+100

Here's a less memory-hungry way to do the same. It possibly is a bit faster too. (but that is difficult to check for me)

My computer did not have sufficient memory to run the first program with those huge files. This one does, but I still needed to create a list of ony tmax's first (the first 1/4 of the file), then print it, and then delete the list in order to have enough memory for nmax's, tmin's and nmin's.

But this one too says the nmin's inside the 2018 file are all -999.0. If that doesn't make sense, could you check what the R-code makes of it then? I suspect that it is just what's in the file. The other possibility is of course, that I got it all wrong (which I doubt). However, I tried the 2017 file too, and that one does not have such problem: all of tmax, nmax, tmin, nmin have around 37% -999.0 's.

Anyway, here's the second code:

import os
import struct

# load_data()
#   data_store : object to append() data items (floats) to
#   num        : number of floats to read and store
#   datafile   : opened binary file object to read float data from
#
def load_data(data_store, num, datafile):
    for i in range(num):
        data = datafile.read(4)  # process one float (=4 bytes) at a time
        item = struct.unpack("<f", data)[0]  # '<' means little endian
        data_store.append(item) 

# save_list() saves a list of float's as strings to a file
#
def save_list(filename, datalist):
    output = open(filename, "wt")
    for item in datalist:
        output.write(str(item) + '\n')
    output.close()

#### MAIN ####

datafile = open('data.bin','rb')

# Get file size so we can calculate number of points without reading
# the (large) file entirely into memory.
#
file_info = os.stat(datafile.fileno())

# Calculate number of points, i.e. number of each tmax's, nmax's,
# tmin's, nmin's. A point is 4 floats of 4 bytes each, hence number
# of points = file-size / (4*4)
#
num = int(file_info.st_size / 16)

tmax_list = list()
load_data(tmax_list, num, datafile)
save_list("tmax.txt", tmax_list)
del tmax_list   # huge list, save memory

nmax_list = list()
load_data(nmax_list, num, datafile)
save_list("nmax.txt", nmax_list)
del nmax_list   # huge list, save memory

tmin_list = list()
load_data(tmin_list, num, datafile)
save_list("tmin.txt", tmin_list)
del tmin_list   # huge list, save memory

nmin_list = list()
load_data(nmin_list, num, datafile)
save_list("nmin.txt", nmin_list)
del nmin_list   # huge list, save memory
2
  • This worked in the sense that a summary table built in Spotfire and in R matched. This helped me figure out that the R code previously used was actually incorrect. Thanks!
    – Mark P.
    Sep 26, 2018 at 13:57
  • Thanks for the +100 bounty!
    – Hkoof
    Sep 30, 2018 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.