How to read a file line by line in PHP, without completely loading it in memory?

My file is too large to open in memory so I always got memory exhaust errors.

The file size is 1Gb.

share|improve this question
    
see my answer at this link – Sohail Ahmed Nov 6 '12 at 7:51
5  
You should use fgets() without $length parameter. – Carlos Nov 6 '12 at 7:51
10  
Would you like to mark as answer on any of the following? – Kim Stacks Apr 17 '14 at 7:33

11 Answers 11

You can use the fgets() function to read the file line by line:

$handle = fopen("inputfile.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // process the line read.
    }

    fclose($handle);
} else {
    // error opening the file.
} 
share|improve this answer
3  
How does this account for the too large to open in memory part? – Starx Nov 6 '12 at 7:53
36  
You are not reading the entire file in memory. The max memory needed to run this depends on the longest line in the input. – codaddict Nov 6 '12 at 7:54
8  
Of course remembering to fclose($handle); too ;) – zelanix Mar 7 '14 at 3:09
8  
This method will still fail to address the original memory problem if the file has really long lines or fails to use the \n character for whatever reason – Brandin Oct 27 '14 at 13:12
6  
@Brandin - Moot - In those situations, the asked question, which is to read a file LINE BY LINE, does not have a well-defined result. – ToolmakerSteve Jun 30 '16 at 18:57
if ($file = fopen("file.txt", "r")) {
    while(!feof($file)) {
        $line = fgets($file);
        # do same stuff with the $line
    }
    fclose($file);
}
share|improve this answer
5  
As @Cuse70 said in his answer this will lead to an infinite loop if the file does not exist or can't be open. Test for if($file) before the while loop – FrancescoMM Apr 15 '15 at 9:18
5  
I know this is old, but: using while(!feof($file)) isn't recommended. Have a look here. – Kevin Van Ryckegem Nov 14 '15 at 22:34
    
BTW: "If there is no more data to read in the file pointer, then FALSE is returned." php.net/manual/en/function.fgets.php ... Just in case – everyman Jan 27 '16 at 10:33
2  
feof() doesn't exist anymore? – futzlarson Jul 15 '16 at 16:39

You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)

<?php

$file = new SplFileObject("file.txt");

// Loop until we reach the end of the file.
while (!$file->eof()) {
    // Echo one line from the file.
    echo $file->fgets();
}

// Unset the file to call __destruct(), closing the file handle.
$file = null;
share|improve this answer
2  
much cleaner solution. thanks ;) haven't used this class yet, there are more interesting functions here to explore: php.net/manual/en/class.splfileobject.php – Lukas Liesis May 10 '15 at 19:29
5  
Thanks. Yes, for example you can add this line before while $file->setFlags(SplFileObject::DROP_NEW_LINE); in order to drop newlines at the end of a line. – elshnkhll Nov 9 '15 at 18:22
    
As far as I can see there is no eof() function in SplFileObject ? – Chud37 Jun 21 '17 at 11:26
1  
Here it is: php.net/manual/en/splfileobject.eof.php – elshnkhll Jun 26 '17 at 18:44
    
Thanks! Also, use rtrim($file->fgets()) to strip trailing newlines for each line string that is read if you don't want them. – racl101 Nov 22 '17 at 23:32

Use buffering techniques to read the file.

$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
    $buffer = fread($source_file, 4096);  // use a buffer of 4KB
    $buffer = str_replace($old,$new,$buffer);
    ///
}
share|improve this answer
1  
this deserves more love, as it will work with huge files, even files that have no carriage returns or exceedingly long lines... – Jimmery Jun 11 '15 at 13:50
    
I wouldn't be surprised if the OP didn't really care about actual lines and just wanted to e.g. serve a download. In that case, this answer is just fine (and what most PHP coders would do anyway). – Álvaro González Jan 19 '16 at 10:46

There is a file() function that returns an array of the lines contained in the file.

foreach(file('myfile.txt') as $line) {
   echo $line. "\n";
}
share|improve this answer
18  
The one GB file would be all read into memory and converted to a more than one GB array... good luck. – FrancescoMM Apr 15 '15 at 9:07
3  
This was not the answer to the question asked, but it does answer the more common question many people have when looking here, so it was still useful, thanks. – pilavdzice Apr 26 '16 at 14:17
1  
file() is very convenient for working with small files. Especially when you want an array() as the end result. – hyphen this Jun 16 '17 at 0:06
    
this is a bad idea with bigger files as whole file is being read to an array at once – Flash Thunder Nov 30 '17 at 19:03
foreach (new SplFileObject(__FILE__) as $line) {
    echo $line;
}
share|improve this answer

If you're opening a big file, you probably want to use Generators alongside fgets() to avoid loading the whole file into memory:

/**
 * @return Generator
 */
$fileData = function() {
    $file = fopen(__DIR__ . '/file.txt', 'r');

    if (!$file)
        die('file does not exist or cannot be opened');

    while (($line = fgets($file)) !== false) {
        yield $line;
    }

    fclose($file);
};

Use it like this:

foreach ($fileData() as $line) {
    // $line contains current line
}

This way you can process individual file lines inside the foreach().

Note: Generators require >= PHP 5.5

share|improve this answer

Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.

share|improve this answer

One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.

$handle = fopen("some_file.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        $line = str_replace("\n", "", $line);
    }
    fclose($handle);
}
share|improve this answer

This how I manage with very big file (tested with up to 100G). And it's faster than fgets()

$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if( $fh = fopen("file.txt", "r") ){ 
            $left='';
    while (!feof($fh)) {// read the file
       $temp = fread($fh, $block);  
       $fgetslines = explode("\n",$temp);
       $lines[0]=$left.$lines[0];
       if(!feof($fh) )$left = array_pop($lines);           
       foreach($lines as $k => $line){
           //do smth with $line
        }
     }
}
fclose($fh);
share|improve this answer

Function to Read with array return

function read_file($filename = ''){
    $buffer = array();
    $source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
    while (!feof($source_file)) {
        $buffer[] = fread($source_file, 4096);  // use a buffer of 4KB
    }
    return $buffer;
}
share|improve this answer
4  
This would create a single array of more than one GB in memory (good luck with it) divided not even in lines but in arbitrary 4096 character chunks. Why on earth would you want to do that? – FrancescoMM Apr 15 '15 at 9:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.