I want to read the contents of a file into a buffer.
What would be the most efficient and portable option?
Portability between Linux and Windows is a headache, since Linux is a POSIX-conformant system with high-quality toolchain for C; whereas, Windows does not.
If you stick to the standard and don't mind a race condition (between getting the file's size and reading the contents), this may suffice:
#include <stdio.h>
#include <stdlib.h>
FILE *f = fopen("textfile.txt", "rb");
fseek(f, 0, SEEK_END);
long fsize = ftell(f);
fseek(f, 0, SEEK_SET); /* same as rewind(f); */
char *string = malloc(fsize + 1);
fread(string, fsize, 1, f);
fclose(f);
string[fsize] = 0;
// use the string, then ...
free(string);
Here string will contain the contents of the text file as a properly 0-terminated C string. This code is just standard C, it's not POSIX-specific (although that it doesn't guarantee it will work/compile on Windows).
If you do care about the race condition, see this other answer.
rewind(f); is equivalent to fseek(f, 0, SEEK_SET); and could be used here instead. Both are part of <stdio.h>
malloc() and fread(). Here, error checking is omitted only for simplicity - do not copypasta this code verbatim into a production code base.
fseek calls with a whence value of SEEK_END", and that "setting the file position indicator to end-of-file, as with fseek(file, 0, SEEK_END), has undefined behavior for a binary stream".
Here is what I would recommend.
It should conform to C89, and be completely portable. In particular, it works also on pipes and sockets on POSIXy systems.
The idea is that we read the input in large-ish chunks (READALL_CHUNK), dynamically reallocating the buffer as we need it. We only use realloc(), fread(), ferror(), and free():
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
/* Size of each input chunk to be
read and allocate for. */
#ifndef READALL_CHUNK
#define READALL_CHUNK 262144
#endif
#define READALL_OK 0 /* Success */
#define READALL_INVALID -1 /* Invalid parameters */
#define READALL_ERROR -2 /* Stream error */
#define READALL_TOOMUCH -3 /* Too much input */
#define READALL_NOMEM -4 /* Out of memory */
/* This function returns one of the READALL_ constants above.
If the return value is zero == READALL_OK, then:
(*dataptr) points to a dynamically allocated buffer, with
(*sizeptr) chars read from the file.
The buffer is allocated for one extra char, which is NUL,
and automatically appended after the data.
Initial values of (*dataptr) and (*sizeptr) are ignored.
*/
int readall(FILE *in, char **dataptr, size_t *sizeptr)
{
char *data = NULL, *temp;
size_t size = 0;
size_t used = 0;
size_t n;
/* None of the parameters can be NULL. */
if (in == NULL || dataptr == NULL || sizeptr == NULL)
return READALL_INVALID;
/* A read error already occurred? */
if (ferror(in))
return READALL_ERROR;
while (1) {
if (used + READALL_CHUNK + 1 > size) {
size = used + READALL_CHUNK + 1;
/* Overflow check. Some ANSI C compilers
may optimize this away, though. */
if (size <= used) {
free(data);
return READALL_TOOMUCH;
}
temp = realloc(data, size);
if (temp == NULL) {
free(data);
return READALL_NOMEM;
}
data = temp;
}
n = fread(data + used, 1, READALL_CHUNK, in);
if (n == 0)
break;
used += n;
}
if (ferror(in)) {
free(data);
return READALL_ERROR;
}
temp = realloc(data, used + 1);
if (temp == NULL) {
free(data);
return READALL_NOMEM;
}
data = temp;
data[used] = '\0';
*dataptr = data;
*sizeptr = used;
return READALL_OK;
}
Above, I've used a constant chunk size, READALL_CHUNK == 262144 (256*1024). This means that in the worst case, up to 262145 chars are wasted (allocated but not used), but only temporarily. At the end, the function reallocates the buffer to the optimal size. Also, this means that we do four reallocations per megabyte of data read.
The 262144-byte default in the code above is a conservative value; it works well for even old minilaptops and Raspberry Pis and most embedded devices with at least a few megabytes of RAM available for the process. Yet, it is not so small that it slows down the operation (due to many read calls, and many buffer reallocations) on most systems.
For desktop machines at this time (2017), I recommend a much larger READALL_CHUNK, perhaps #define READALL_CHUNK 2097152 (2 MiB).
Because the definition of READALL_CHUNK is guarded (i.e., it is defined only if it is at that point in the code still undefined), you can override the default value at compile time, by using (in most C compilers) -DREADALL_CHUNK=2097152 command-line option -- but do check your compiler options for defining a preprocessor macro using command-line options.
fstat() approach, by the way.) Reading the stream until read fails is really the only portable option that works on everything you can get a FILE handle on. I'd prefer new C programmers to know that before it bites them in the ankle, you see.
Commented
Aug 13, 2018 at 14:30
nftw()/fts_..()/glob()/wordexp() rather than opendir()/readdir()/closedir(), to easily handle files/directories being added/deleted/renamed during traversal. I know I should not care, but I really don't like the idea of more C programmers writing code that works only in specific circumstances, and silently fails - or worse yet, destroys data - otherwise. The world is already full of such code, and we need less of it, not more.
Commented
Aug 14, 2018 at 1:31
n < READALL_CHUNK (although this probably as little to no performance impact). This is mentioned in this QA. Am I right ?
Commented
Aug 28, 2019 at 20:43
A portable solution could use getc.
#include <stdio.h>
char buffer[MAX_FILE_SIZE];
size_t i;
for (i = 0; i < MAX_FILE_SIZE; ++i)
{
int c = getc(fp);
if (c == EOF)
{
buffer[i] = 0x00;
break;
}
buffer[i] = c;
}
If you don't want to have a MAX_FILE_SIZE macro or if it is a big number (such that buffer would be to big to fit on the stack), use dynamic allocation.
fread() function provides something more efficient.
fgets could also do the trick). Also, I don't like to use POSIX functions such as fread on Windows, because the POSIX implementation by this operating system is often different from the specifications. About heap allocation, it is written at this end of my answer.
fread() is not POSIX-specific. If you don't like to use it, you may abandon fgetc() as well.