2

I have a text file(unsigned short values) as follows

abc.txt

2311
1231
1232
54523
32423

I'm reading this file in my function using while loop and storing in a buffer as follows

while(!feof(ref))
{
    fscanf(ref,"%d\n",&ref[count]);
    count++;
}

It is taking too much time for reading large file is there any way to optimize the fscanf operation.

5
  • 6
    Don't do while(!feof(ref)), it will not work as you expect. The reason is that the EOF flag is not set until after you try to read from beyond the end of the file. that means you will loop once to many, with a fscanf that will fail without you noticing. Instead, in your case, do while (fscanf(ref, "%d", &ref[count]) == 1). That will loop as long the the file is open, or until there is an error in the file. Feb 26, 2014 at 14:42
  • Define "too much time". For example, if it is still too much time just to read the file (and not do any decoding) -- game over. Feb 26, 2014 at 14:44
  • You really don't want the trailing newline in a format, especially if a human will ever be providing the input. It means 'skip white space of any sort, including newlines, until you read a something other than white space'. So, if you were a human providing input, you'd type the number, then a newline, and then the program would wait for some other character to be typed — maybe the e in 'expletive!', or maybe an interrupt or quit signal. Feb 26, 2014 at 14:50
  • @ScottHunter Actually i'm comparing this values with another text file to find out matching pattern. i.e. TextFile1 contains 100000 datas and textfile2 contains 10 data's. I want to find exact positions in which the textfile1 is matching with textfile2.
    – kani
    Feb 26, 2014 at 15:01
  • ref ,Duplicate names.
    – BLUEPIXY
    Feb 26, 2014 at 15:14

4 Answers 4

1

This is because secondary memory access is slower than primary memory access. First dump the file into primary memory using fread() in binary mode. Then read from primary memory integer by integer.

1

A common way is to read a larger chunk into a large memory buffer, and then parse out the data from that buffer.

Another way may be to instead memory map the file, then the OS will put the file into your process virtual memory map, so you can read it like reading from memory.

2
  • Aren't the C library functions for file handling already buffered?
    – Ali
    Feb 26, 2014 at 16:08
  • @Ali Indeed they are, and the buffer is often synchronized with the filesystems on-disk block size. However, it may be faster to read larger chunks into memory and then loop over the memory, than doing many smaller reads and using fscanf (which is a very generic parser and may be less effective than one made specifically for the file format in question). Feb 26, 2014 at 16:19
0

use a local buffer and read blocks of data using fread() in binary mode. Parse your text data and continue with the next block.

tune your buffer size properly, maybe 64K or 1Mb in size, it depends on your application.

#include <stdio.h>

int BUFFER_SIZE = 1024;
FILE *source;
FILE *destination;
int n;
int count = 0;
int written = 0;

int main()
{
    unsigned char buffer[BUFFER_SIZE];

    source = fopen("myfile", "rb");

    if (source)
    {
        while (!feof(source))
        {
            n = fread(buffer, 1, BUFFER_SIZE, source);
            count += n;
            // here parse data
        }
    }

    fclose(source);
    return 0;
}
3
  • Ouch! See right below the answer why while(!feof(ref)) is a bad idea. I am not sure I see why this will help. As far as I know, the C library functions are supposed to be / are usually buffered. So your buffer would be the third, superfluous buffer, slowing things down.
    – Ali
    Feb 26, 2014 at 16:05
  • 1
    This does not answer the question directly; the OP was about reading in unsigned short values, your read-buffer-by-buffer will make the "here parse data" tricky as each end-of-buffer may bisect a text line. In any case setvbuf() would be a lot simpler.
    – TonyWilk
    Feb 26, 2014 at 18:54
  • when it comes to parse text file efficiently you want always treat it as a binary file irrespective of the text formatting. Then, parse your raw data inside the buffer. The question was about making the reading faster Feb 27, 2014 at 8:35
0

This may be faster if each line has only one number, atoi() is a lot faster than using fscanf()

#define BUFLEN 128
#define ARRAY_SIZE 12345
int myarray[ARRAY_SIZE];
char buffer[BUFLEN]
FILE *fp= fopen(...);
index=0;
do
{
    if( fgets(buffer, BUFLEN-1, fp) == NULL )
       break;
    myarray[index++]= atoi(buffer);
    if( index >= ARRAY_SIZE)
        break;

}while(!feof(fp));

...hastily typed in code, not compiled or run ;)

You can improve the file reading by setting a stream buffer e.g.

#define STRMBUF_SIZE (64*1024)
char strmbuf[STRMBUF_SIZE];
setvbuf( fp, strmbuf,_IOFBF,STRMBUF_SIZE);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.