3

My goal is: Open a binary file, store (append) every 100 values into this file, and continue doing so until finishing. To do so, I use the following simple loop to simulate that:

import numpy as np
import random

alist=[]
c = 1

for i in range(1000):
    alist.append(i)
    if i == (c*100):
        np.array(alist).tofile("file.bin")
        print alist
        c = c + 1
        alist[:] = [] # clear the list before continuing

However, when I check the size of file.bin, I feel numpy does not append rather it is replacing which is not what I want. How to fix that?

Thank you.

1
  • Yes, np.ndarray.tofile overwrites the file on each invocation. Jul 20, 2017 at 18:50

1 Answer 1

4

Of course numpy is replacing/overwriting your old data every time you (re-)open the file for writing. This is almost universal behavior of most tofile() like functions (and not only in numpy).

Solution: Open a file handle for writing before the loop and pass that to tofile() function. Like this:

import numpy as np
import random

alist=[]
c = 1
with open("file.bin", "wb") as f: # or choose 'w+' mode - read "open()" documentation
    for i in range(1000):
        alist.append(i)
        if i == (c*100):
            np.array(alist).tofile(f)
            print alist
            c = c + 1
            alist[:] = [] # clear the list before continuing

Now the code opens the file before entering the loop and tofile() method re-uses an already opened file handle instead of re-opening and thus overwriting an existing file (created in the run of the loop).

2
  • Kindly, may I know how to implement your solution? Thank you
    – Kristofer
    Jul 20, 2017 at 18:51
  • Thank you very much. Appreciate it
    – Kristofer
    Jul 20, 2017 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.