3

I'm trying to parse a 50 megabyte .csv file. The file itself is fine, but I'm trying to get past the massive timeout issues involved. Every is set upload wise, I can easily upload and re-open the file but after the browser timeout, I receive a 500 Internal error.

My guess is I can save the file onto the server, open it and keep a session value of what line I dealt with. After a certain line I reset the connect via refresh and open the file at the line I left off with. Is this a do-able idea? The previous developer made a very inefficient MySQL class and it controls the entire site, so I don't want to write my own class if I don't have to, and I don't want to mess with his class.

TL;DR version: Is it efficient to save the last line I'm currently on of a CSV file that has 38K lines of products then, and after X number of rows, reset the connection and start from where I left off? Or is there another way to parse a Large CSV file without timeouts?

NOTE: It's the PHP script execution time. Currently at 38K lines, it takes about 46 minutes and 5 seconds to run via command line. It works correctly 100% of the time when I remove it from the browser, suggesting that it is a browser timeout. Chrome's timeout is not editable as far as Google has told me, and Firefox's timeout works rarely.

5
  • You should elaborate on what's timing out. Your connection to MySQL, the PHP script execution time, or something else? Have you considered setting the timeout longer?
    – JYelton
    Aug 3, 2012 at 17:01
  • You should consider having the script splitting the file at first. Aug 3, 2012 at 17:09
  • @JYelton I It's the PHP script execution time. Currently at 38K lines, it takes about 46 minutes and 5 seconds to run via command line. It works correctly 100% of the time when I remove it from the browser, suggesting that it is a browser timeout. Chrome's timeout is not editable as far as Google has told me, and Firefox's timeout works rarely.
    – lxndr
    Aug 3, 2012 at 17:13
  • 2
    46 minutes to do <50MB of work?! I'm suspicious that you're not doing what you think you're doing. If you're trying to upload that large of a file for processing you should really just send the whole thing as a zip to the server, then processes it there and return the results. This can also be done as a backend transfer and won't timeout the browser. Sending line by line is painful and I am guessing you're bottleneck is from you using an interface meant for KB of data instead of 50MB.
    – Pyrce
    Aug 3, 2012 at 17:19
  • Looking over the old developers' code, I'm noticing that they use about 15 if...else statements, 7 functions (each with at least 10 if...else statements), and each function does some really inefficient MySQL Queries. I completely agree that the interface was meant for KB of data, but unfortunately I have a time crunch that says "quick or dirty, we don't care. find the fix before you leave today." So I think I'll give my idea a shot and I'll reply with the results. Thanks for everyones' ideas.
    – lxndr
    Aug 3, 2012 at 17:29

3 Answers 3

3

You could do something like this:

<?php

namespace database;

class importcsv
{
    private $crud;

    public function __construct($dbh, $table)
    {
        $this->crud = new \database\crud($dbh, $table);
        return $this;
    }

    public function import($columnNames, $csv, $seperator)
    {
        $lines = explode("\n", $csv);

        foreach($lines as $line)
        {
            \set_time_limit(30);
            $line = explode($seperator, $line);

            $data = new \stdClass();

            foreach($line as $i => $item) 
            {
                if(isset($columnNames[$i])&&!empty($columnNames[$i]))
                    $data->$columnNames[$i] = $item;
            }

            @$x++;

            $this->crud->create($data);
        }

        return $x;
    }

    public function importFile($columnNames, $csvPath, $seperator)
    {
        if(file_exists($csvPath))
        {
            $content = file_get_contents($csvPath);
            return $this->import($columnNames, $content, $seperator);
        }
        else
        {
            // Error
        }
    }
}

TL;DR: \set_time_limit(30); everytime you loop throu a line might fix your timeout issues.

2
  • This is a decent piece of code and it would work if it wasn't for the fact that my products have a bunch of different things that need to happen instead of just doing an insert. Thanks, and I may take the code and implement it for other ideas.
    – lxndr
    Aug 3, 2012 at 18:00
  • Didn't know set_timeout resets the time. You just saved me many hours of work! :)
    – Inc33
    Sep 7, 2020 at 5:36
2

I suggest to run php from command line and set it as a cron job. This way you don't have to modify your code. There will be no timeout issue and you can easily parse large CSV files.

also check this link

3
  • I wish I can do this option, but it has to be ran on a page and has to be done by people who have absolutely no idea how a website works. I can't give them command line access and ask them to run it once a week because they'll tell me no and add some other colorful words to their refusal.
    – lxndr
    Aug 3, 2012 at 17:22
  • I changed my accepted answer to this one. It is precisely what I did. I sent the package to the server and ran it as a cron job. Runs smoothly, after a 12 hour session. That 50MB file turned into a 256MB file and it'd take 1hr at a time to parse. Thanks for your help.
    – lxndr
    Jan 17, 2013 at 21:30
  • your links gives 404 webpronews.com/php-cli-and-cron-2004-04 Sep 19, 2016 at 5:09
2

Your post is a little unclear due to the typos and grammar, could you please edit?

If you are saying that the Upload itself is okay, but the delay is in processing of the file, then the easiest thing to do is to parse the file in parallel using multiple threads. You can use the java built-in Executor class, or Quartz or Jetlang to do this.

  • Find the size of the file or number of lines.
  • Select a Thread load (Say 1000 lines per thread)
  • Start an Executor
  • Read the file in a loop.
  • For ach 1000 lines, create a Runnable and load it to the Executor
  • Start the Executor
  • Wait till all threads are finished

Each runnable does this:

  1. Fetch a connection
  2. Insert the 1000 lines
  3. Log the results
  4. Close the connection

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.