3

I have to use Perl on a Windows environment at work, and I need to be able to find out the number of rows that a large csv file contains (about 1.4Gb). Any idea how to do this with minimum waste of resources?

Thanks

PS This must be done within the Perl script and we're not allowed to install any new modules onto the system.

1

6 Answers 6

14

Do you mean lines or rows? A cell may contain line breaks which would add lines to the file, but not rows. If you are guaranteed that no cells contain new lines, then just use the technique in the Perl FAQ. Otherwise, you will need a proper CSV parser like Text::xSV.

2
  • 1
    You should amend your question, since every other comment is just doing line counting.
    – jiggy
    Apr 15, 2009 at 15:58
  • +1, good point, but it's also worth mentioning that there is no "official" CSV format -- just a collection of loosely-defined, somewhat incompatible formats that disagree on things like how to quote commas and whether line breaks are allowed in cells. Many tools assume 1 row == 1 line. Apr 15, 2009 at 17:11
9

Yes, don't use perl.

Instead use the simple utility for counting lines; wc.exe

It's part of a suite of windows utilities ported from unix originals.

http://unxutils.sourceforge.net/

For example;

PS D:\> wc test.pl
     12      26     271 test.pl
PS D:\>

Where 12 == number of lines, 26 == number of words, 271 == number of characters.

If you really have to use perl;

D:\>perl -lne "END{print $.;}" < test.pl
12
11
  • 1
    Sure, wc would be the way to go on *nix where it will already be installed -- but is it really worth downloading a separate executable to do something that takes short line of Perl? Apr 15, 2009 at 15:31
  • Yes, because Cygwin is a must-have for any Windows dev environment.
    – KenE
    Apr 15, 2009 at 15:33
  • 1
    This isn't Cygwin but still a must-have. Counting lines in a file is such a common activity that its definitely worth having this utility.
    – Ed Guiness
    Apr 15, 2009 at 15:34
  • @KenE: Is that sarcasm? FTR UnxUtils are non-Cygwin-based. Apr 15, 2009 at 15:35
  • @edg: I see you added a Perl suggestion, +1. Apr 15, 2009 at 15:36
4
perl -lne "END { print $. }" myfile.csv

This only reads one line at a time, so it doesn't waste any memory unless each line is enormously long.

2
  • Lines are not the same thing as CSV rows. Consider fields with embedded newlines, for instance. Apr 16, 2009 at 16:36
  • @brian: That's true. But it's also true that working with CSV files containing fields with embedded newlines is destined to be painful, because there's no universal agreement across tools on whether or how such things should be encoded -- unfortunately, CSV is not a "standard." Apr 16, 2009 at 17:44
3

This one-liner handles new lines within the rows:

  1. Considering lines with an odd number of quotes.
  2. Considering that doubled quotes is a way of indicating quotes within the field.
  3. It uses the awesome flip-flop operator.

    perl -ne 'BEGIN{$re=qr/^[^"]*(?:"[^"]*"[^"]*)*?"[^"]*$/;}END{print"Count: $t\n";}$t++ unless /$re/../$re/'
    

Consider:

  • wc is not going to work. It's awesome for counting lines, but not CSV rows
  • You should install--or fight to install--Text::CSV or some similar standard package for proper handling.
  • This may get you there, nonetheless.


EDIT: It slipped my mind that this was windows:

perl -ne "BEGIN{$re=qr/^[^\"]*(?:\"[^\"]*\"[^\"]*)*?\"[^\"]*$/;}END{print qq/Count: $t\n/;};$t++ unless $pq and $pq = /$re/../$re/;"

The weird thing is that The Broken OS' shell interprets && as the OS conditional exec and I couldn't do anything to change its mind!! If I escaped it, it would just pass it that way to perl.

0

Upvote for edg's answer, another option is to install cygwin to get wc and a bunch of other handy utilities on Windows.

3
  • IME, Cygwin adds too much complication unless you want to run a pseudo-unix environment. MinGW and MSYS provide a lighter weight system that works well for software development. For simple command line tools, GnuWin32 offers a good selection of tools with low impact, simple installers.
    – daotoad
    Apr 15, 2009 at 16:26
  • Thanks for the tip - I'll give those a try sometime!
    – KenE
    Apr 15, 2009 at 17:13
  • wc is not the answer since it counts lines, which is not the same as a CSV row. See Axeman's answer. Apr 16, 2009 at 16:35
-1

I was being idiotic, the simple way to do it in the script is:

open $extract, "<${extractFileName}" or die ("Cannot read row count of $extractFileName");
$rowCount=0;    
while (<$extract>)
{
    $rowCount=$rowCount+1;
}

close($extract);
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.