17

I want to read through a binary file. Googling "python binary eof" led me here.

Now, the questions:

  1. Why does the container (x in the SO answer) contain not a single (current) byte but a whole bunch of them? What am I doing wrong?
  2. If it should be so and I am doing nothing wrong, HOW do read a single byte? I mean, is there any way to detect EOF while reading the file with read(1) method?
2
  • 1
    I dunno why can't I harmlessly get a little help from community? Why the questions that don't deserve a 5-page answer, a wiki page, or a place in the glorious hall of fame for great ones will ALWAYS get downvoted? Why should I ask only about rocket-science?
    – mekkanizer
    Commented Aug 23, 2014 at 20:05
  • If you're not satisfied of the way you question are perceived here, a comment in not the appropriate reaction as the downvoter has gone long time ago -- and won't probably read that comment. But other peoples will. If you really disagree and are absolutely certain that your questions reach the quality standard expected here, you might raise your case on meta. But be prepared before doing so. Commented Aug 23, 2014 at 21:03

4 Answers 4

19

To quote the documentation:

file.read([size])

Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes). If the size argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a string object. An empty string is returned when EOF is encountered immediately. (For certain files, like ttys, it makes sense to continue reading after an EOF is hit.) Note that this method may call the underlying C function fread() more than once in an effort to acquire as close to size bytes as possible. Also note that when in non-blocking mode, less data than was requested may be returned, even if no size parameter was given.

That means (for a regular file):

  • f.read(1) will return a byte object containing either 1 byte or 0 byte is EOF was reached
  • f.read(2) will return a byte object containing either 2 bytes, or 1 byte if EOF is reached after the first byte, or 0 byte if EOF in encountered immediately.
  • ...

If you want to read your file one byte at a time, you will have to read(1) in a loop and test for "emptiness" of the result:

# From answer by @Daniel
with open(filename, 'rb') as f:
    while True:
        b = f.read(1)
        if not b:
            # eof
            break
        do_something(b)

If you want to read your file by "chunk" of say 50 bytes at a time, you will have to read(50) in a loop:

with open(filename, 'rb') as f:
    while True:
        b = f.read(50)
        if not b:
            # eof
            break
        do_something(b) # <- be prepared to handle a last chunk of length < 50
                        #    if the file length *is not* a multiple of 50

In fact, you may even break one iteration sooner:

with open(filename, 'rb') as f:
    while True:
        b = f.read(50)
        do_something(b) # <- be prepared to handle a last chunk of size 0
                        #    if the file length *is* a multiple of 50
                        #    (incl. 0 byte-length file!)
                        #    and be prepared to handle a last chunk of length < 50
                        #    if the file length *is not* a multiple of 50
        if len(b) < 50:
            break

Concerning the other part of your question:

Why does the container [..] contain [..] a whole bunch of them [bytes]?

Referring to that code:

for x in file:  
   i=i+1  
   print(x)  

To quote again the doc:

A file object is its own iterator, [..]. When a file is used as an iterator, typically in a for loop (for example, for line in f: print line.strip()), the next() method is called repeatedly. This method returns the next input line, or raises StopIteration when EOF is hit when the file is open for reading (behavior is undefined when the file is open for writing).

The the code above read a binary file line-by-line. That is stopping at each occurrence of the EOL char (\n). Usually, that leads to chunks of various length as most binary files contains occurrences of that char randomly distributed.

I wouldn't encourage you to read a binary file that way. Please prefer one a solution based on read(size).

5
  • There are so many answers close to this that DO NOT WORK! thank you for posting a great and detailed answer!!! I could not have done my project without this answer!
    – Noah Gary
    Commented Apr 3, 2016 at 21:10
  • For the f.read(1) example: What if your byte stream has a zero byte in it, then the test would exit early before you had finished reading the file Commented Jun 12, 2017 at 14:28
  • 1
    @slashdottir In Python, only empty sequences are False (docs.python.org/3.1/library/stdtypes.html) And whatever is not False is True. So, if your byte string contains \x00, it is not empty, hence b"\x00" is True. You can test that in the REPL: "T" if b"\x00" else "F" Commented Jun 12, 2017 at 14:56
  • Word to the wise here, it's important to realize what's going on under the hood when buffering files. There are actually several buffer objects being created that contain parts of the buffered file. See here. Python's supposed to make this unlikely, but the classic problem raised by this is that you end up with part of your file still in one of those underlying buffer objects when done. Be sure to flush them if your missing data. Commented Aug 7, 2018 at 19:03
  • I don't love the break statement. It voids the guarantee the while's predicate is false when it exits. Commented Sep 23, 2020 at 23:33
4

"" will signify the end of the file

with open(filename, 'rb') as f:
    for ch in iter(lambda: f.read(1),""): # keep calling f.read(1) until end of the data
        print ch
1
  • I think for python3.x it would have to be: ... for ch in iter(lambda: f.read(1), b""): ... Commented Jun 6, 2023 at 15:57
1

Reading byte-by-byte:

with open(filename, 'rb') as f:
    while True:
        b = f.read(1)
        if not b:
            # eof
            break
        do_something(b)
2
  • but if your file contains a zero byte, this will end early before file is completely read Commented Jun 12, 2017 at 4:46
  • @slashdottir: wrong. not '\0' is False. Only not '' is True.
    – Daniel
    Commented Jun 13, 2017 at 19:50
1

Here is what I did. The call to read returns a falsy value when it encounters the end of the file, and this terminates the loop. Using while ch != "": copied the image but it gave me a hung loop.

from sys import argv
donor = argv[1]
recipient = argv[2]
# read from donor and write into recipient
# with statement ends, file gets closed
with open(donor, "rb") as fp_in:
    with open(recipient, "wb") as fp_out:
        ch = fp_in.read(1)
        while ch:
            fp_out.write(ch)
            ch = fp_in.read(1)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.