5

I have some code that uses the low level i/o read and write system calls, as described on page 170 of the C programming language book Kernighan and Ritchie. The function prototypes are this

int n_read = read  ( int fd, char *buf, int n )
int n_read = write ( int fd, char *buf, int n )

now the two .c file that uses these read and write are called by a larger fortran based program to read and write lots of data.

the C code is simply this, with no #include of any kind, having the underscore after the function name and passing by reference:

int read_ ( int *descriptor, char *buffer, int *nbyte )
{
   return ( read( *descriptor, buffer, *nbyte ) );
}

int write_ ( int *descriptor, char *buffer, int *nbyte )
{
   return ( write( *descriptor, buffer, *nbyte ) );
}

and the larger fortran based program will do something like this

INTEGER nbyte
COMPLEX*16 matrix(*)
INTEGER READ, WRITE
EXTERNAL READ, WRITE

status = READ( fd, matrix, nbyte )
if ( status .eq. -1 ) then
   CALL ERROR('C call read failure')
   stop
endif

As you may have already guessed, this works fine for nbyte values less than 2^31. I have a need to read more than 2 GB of data, so i need nbyte to be a long integer and INTEGER*8 in fortran.

Is there an equivalent read64 and write64, like there is an lseek64 provided by unistd.h and features.h ?

what is the best way to recode this? should i use fread and fwrite ? is the int fd from the low level write the same as FILE *stream from fread() ?

my requirement is being able to pass a long integer of 8 bytes to allow for values up to 100 to 500 gigabytes or an integer having 12 digits, which is all for the value of nbyte

Am i gaining anything or losing out by currently using read and write which is identified as a "system call" ? What does this mean?

4
  • 3
    What about successive reads in small chunks until all is read? Commented Apr 13, 2016 at 19:09
  • K&R C is not something you should learn nowadays, except if you are curious for historic reasons. Lots of stuff there is frowned upon, outdated or plain wrong nowadays. For example, if you had just RTFM for the read() function, you would have found the declaration that Leandros gave you and probably solved your problem yourself. Commented Apr 13, 2016 at 19:21
  • That would at best be an ugly workaround, @WeatherVane. It is still something to keep in mind if the Fortran side can't interface with C code. Commented Apr 13, 2016 at 19:23
  • For up-to-date information on library functions, open a terminal window, and use the man command. For example, man 2 read tells you all about the read function. The number 2 indicates the section in the manual. Section 1 is mostly shell commands. Sections 2 and 3 have the C library functions. Commented Apr 13, 2016 at 19:26

2 Answers 2

5

Edit: You can't, at least not on Linux. read will never transfer more than what a 32-bit integer can hold.

From the manpages of Linux on read:

On Linux, read() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.)

This is not a contraint of POSIX, it's allowed by POSIX, but in the end it's implementation defined how read behaves. As Andrew Hanle reports, reading a 32GB file works just fine on Solaris. In this case, my old answer is still valid.

Old Answer:

read can work with 64-bit files just fine. It's defined in <unistd.h> as the following:-

ssize_t read(int fd, void *buf, size_t count);

You would have to adjust your routines to work with size_t instead of int, to properly support big files.

You should check SSIZE_MAX (the maximum value supported for count), before using read with a big file, and abort if it's to small (or split into smaller chunks). SSIZE_MAX is an implementation defined value.

9
  • 1
    How do you know that size_t is big enough to handle 64-bit numbers? It is allowed to have a maximum representable value as small as 65534. Commented Apr 13, 2016 at 19:22
  • 1
    @JohnBollinger He's using the POSIX API, not stdio. Doesn't POSIX require size_t to be large enough for the maximum file size?
    – Barmar
    Commented Apr 13, 2016 at 19:27
  • @Barmar, POSIX and C both allow SIZE_MAX to be as small as 65534. See pubs.opengroup.org/onlinepubs/009695399/basedefs/stdint.h.html Commented Apr 13, 2016 at 19:29
  • Ahh, I see. POSIX has off_t for file offsets, but a single read() may still be limited to size_t.
    – Barmar
    Commented Apr 13, 2016 at 19:30
  • @JohnBollinger If you can't assume that size_t is 64 bits on a 64-bit system, then size_t is rather useless, isn't it? (Perhaps that was your point.) In any case, the OP could check the sizeof(size_t) at runtime, and abort if it's not 8. Commented Apr 13, 2016 at 19:32
0

As @Leandros observed, POSIX-conforming implementations of read() and write() accept byte counts of type size_t, and return byte counts of type ssize_t. These are probably the definitions that actually apply to you, as the read() and write() functions are not specified by the C standard. That's a distinction without much difference, however, because size_t is not required to be wider than int -- in fact, it can be narrower.

You anyway have a bigger problem. The Fortran code seems to assume that the C functions it is calling will read / write the full specified number of bytes or else fail, but POSIX read() and write() are not guaranteed to do that when they succeed. Indeed, there was a question around here the other day that hinged on the fact that these functions did not transfer more bytes at a time than can be represented by a signed, 32-bit integer, even on a 64-bit system with 64-bit [s]size_t.

You can kill both of these birds with one stone by implementing the read_() and write_() functions to loop, performing successive calls to the underlying read() or write() function, until the full number of specified bytes is transferred or an error occurs.

7
  • You have to. read does not support reading more than 0x7ffff000 bytes.
    – Leandros
    Commented Apr 13, 2016 at 19:43
  • @Leandros, whereas that is true of some implementations, which is already covered in my answer, POSIX does not require it to be true of all implementations. But yes, that makes it a very good idea to loop. Commented Apr 13, 2016 at 19:50
  • Correct. You can't generally answer it, in a non implementation defined manner. While it's not possible on Linux, some other POSIX compliant platforms might support it. (thinking about Darwin here)
    – Leandros
    Commented Apr 13, 2016 at 19:55
  • 1
    @Leandros You have to. read does not support reading more than 0x7ffff000 bytes. Not technically true: If the value of nbyte is greater than {SSIZE_MAX}, the result is implementation-defined. SSIZE_MAX on a 64-bit system is quite a bit larger than 0x7ffff000. I just ran a quick test on Solaris 11 and successfully read 32GB in one read operation and only couldn't read larger because a 64GB allocation failed. Good code will handle short reads, though, no matter how large the requested number of bytes may be. Commented Apr 13, 2016 at 20:13
  • 2
    @Leandros Well, your comment that I was responding to didn't qualify your original "read does not support reading more than 0x7ffff000 bytes". It's irrelevant, though, because code must handle short reads anyway: If a read() is interrupted by a signal after it has successfully read some data, it shall return the number of bytes read. Commented Apr 13, 2016 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.