170

I'm using python (Django Framework) to read a CSV file. I pull just 2 lines out of this CSV as you can see. What I have been trying to do is store in a variable the total number of rows the CSV also.

How can I get the total number of rows?

file = object.myfilePath
fileObject = csv.reader(file)
for i in range(2):
    data.append(fileObject.next()) 

I have tried:

len(fileObject)
fileObject.length
6
  • 1
    What is file_read? Is it a file handle (as in file_read = open("myfile.txt")? Apr 19, 2013 at 15:50
  • 1
    file_read = csv.reader(file) updated question should make sense now.
    – GrantU
    Apr 19, 2013 at 15:50
  • Have a look at this question for thoughts on that topic: stackoverflow.com/questions/845058/…
    – shredding
    Apr 19, 2013 at 15:53
  • 1
  • 1
    The accepted answer by @martjin-pieters is correct, but this question is worded poorly. In your pseudocode, you almost certainly want to count the number of rows i.e. records – as opposed to "Count how many lines are in a CSV". Because some CSV datasets may include fields which may be multiline.
    – dancow
    Aug 19, 2020 at 4:07

21 Answers 21

248

You need to count the number of rows:

row_count = sum(1 for row in fileObject)  # fileObject is your csv.reader

Using sum() with a generator expression makes for an efficient counter, avoiding storing the whole file in memory.

If you already read 2 rows to start with, then you need to add those 2 rows to your total; rows that have already been read are not being counted.

10
  • 3
    Thanks. That will works, but do I have to read the lines first? That seems a bit of a hit?
    – GrantU
    Apr 19, 2013 at 15:54
  • 9
    You have to read the lines; the lines are not guaranteed to be a fixed size, so the only way to count them is to read them all.
    – Martijn Pieters
    Apr 19, 2013 at 15:55
  • 1
    @Escachator: what platform are you on? Are there EOF (CTRL-Z, \x1A) characters in the file? How did you open the file?
    – Martijn Pieters
    Apr 10, 2015 at 21:45
  • 4
    @Escachator: Your filename has 53 characters then. The reader takes an iterable or an open file object but not a filename.
    – Martijn Pieters
    Apr 10, 2015 at 21:51
  • 15
    Note that if you want to then iterate through the reader again (to process the rows, say) then you'll need to reset the iterator, and recreate the reader object: file.seek(0) then fileObject = csv.reader(file) Jul 12, 2018 at 22:05
94

2018-10-29 EDIT

Thank you for the comments.

I tested several kinds of code to get the number of lines in a csv file in terms of speed. The best method is below.

with open(filename) as f:
    sum(1 for line in f)

Here is the code tested.

import timeit
import csv
import pandas as pd

filename = './sample_submission.csv'

def talktime(filename, funcname, func):
    print(f"# {funcname}")
    t = timeit.timeit(f'{funcname}("{filename}")', setup=f'from __main__ import {funcname}', number = 100) / 100
    print('Elapsed time : ', t)
    print('n = ', func(filename))
    print('\n')

def sum1forline(filename):
    with open(filename) as f:
        return sum(1 for line in f)
talktime(filename, 'sum1forline', sum1forline)

def lenopenreadlines(filename):
    with open(filename) as f:
        return len(f.readlines())
talktime(filename, 'lenopenreadlines', lenopenreadlines)

def lenpd(filename):
    return len(pd.read_csv(filename)) + 1
talktime(filename, 'lenpd', lenpd)

def csvreaderfor(filename):
    cnt = 0
    with open(filename) as f:
        cr = csv.reader(f)
        for row in cr:
            cnt += 1
    return cnt
talktime(filename, 'csvreaderfor', csvreaderfor)

def openenum(filename):
    cnt = 0
    with open(filename) as f:
        for i, line in enumerate(f,1):
            cnt += 1
    return cnt
talktime(filename, 'openenum', openenum)

The result was below.

# sum1forline
Elapsed time :  0.6327946722068599
n =  2528244


# lenopenreadlines
Elapsed time :  0.655304473598555
n =  2528244


# lenpd
Elapsed time :  0.7561274056295324
n =  2528244


# csvreaderfor
Elapsed time :  1.5571560935772661
n =  2528244


# openenum
Elapsed time :  0.773000013928679
n =  2528244

In conclusion, sum(1 for line in f) is fastest. But there might not be significant difference from len(f.readlines()).

sample_submission.csv is 30.2MB and has 31 million characters.

6
  • Should you also close the file? to save space? Nov 21, 2017 at 9:07
  • 1
    Why do you prefer sum() over len() in your conclusion? Len() is faster in your results!
    – gosuto
    Jan 14, 2018 at 8:21
  • 1
    Nice answer. One addition. Although slower, one should prefer the for row in csv_reader: solution when the CSV is supposed to contain valid quoted newlines according to rfc4180. @dixhom how large was the file you've tested?
    – Simon Lang
    Jan 23, 2018 at 14:07
  • Nice one. sum1forline could be even faster if the file is opened as 'rb'.
    – s3dev
    Jan 21, 2021 at 14:26
  • No need to store the iterated value:sum(1 for _ in f) is more memory efficient and produces the same result. Oct 31, 2023 at 19:50
20

To do it you need to have a bit of code like my example here:

file = open("Task1.csv")
numline = len(file.readlines())
print (numline)

I hope this helps everyone.

2
  • 2
    I like this short answer, but it is slower than Martijn Pieters's. For 10M lines, %time sum(1 for row in open("df_data_raw.csv")) cost 4.91s while %time len(open("df_data_raw.csv").readlines()) cost 14.6s. May 13, 2018 at 9:13
  • 1
    The original title to the question ("Count how many lines are in a CSV Python") was worded confusingly/misleadingly, since the questioner seems to want the number of rows/records. Your answer would give a wrong number of rows in any dataset in which there are fields with newline characters
    – dancow
    Aug 19, 2020 at 4:10
14

Several of the above suggestions count the number of LINES in the csv file. But some CSV files will contain quoted strings which themselves contain newline characters. MS CSV files usually delimit records with \r\n, but use \n alone within quoted strings.

For a file like this, counting lines of text (as delimited by newline) in the file will give too large a result. So for an accurate count you need to use csv.reader to read the records.

11

After iterating the whole file with csv.reader() method, you have the total number of lines read, via instance variable line_num:

import csv
with open('csv_path_file') as f:
    csv_reader = csv.reader(f)
    for row in csv_reader:
        pass
    print(csv_reader.line_num)

Quoting the official documentation:

csvreader.line_num

The number of lines read from the source iterator.

Small caveat:

  • total number of lines, includes the header, if the CSV has.
0
10

First you have to open the file with open

input_file = open("nameOfFile.csv","r+")

Then use the csv.reader for open the csv

reader_file = csv.reader(input_file)

At the last, you can take the number of row with the instruction 'len'

value = len(list(reader_file))

The total code is this:

input_file = open("nameOfFile.csv","r+")
reader_file = csv.reader(input_file)
value = len(list(reader_file))

Remember that if you want to reuse the csv file, you have to make a input_file.fseek(0), because when you use a list for the reader_file, it reads all file, and the pointer in the file change its position

6

row_count = sum(1 for line in open(filename)) worked for me.

Note : sum(1 for line in csv.reader(filename)) seems to calculate the length of first line

1
  • 4
    The first one is counting the number of lines in a file. If your csv has line breaks in strings, it wont show accurate results Nov 29, 2018 at 22:20
4

This works for csv and all files containing strings in Unix-based OSes:

import os

numOfLines = int(os.popen('wc -l < file.csv').read()[:-1])

In case the csv file contains a fields row you can deduct one from numOfLines above:

numOfLines = numOfLines - 1
1
  • This is very handy for integrating into a python script. +1
    – Vitalis
    Aug 15, 2020 at 18:16
4

I think we can improve the best answer a little bit, I'm using:

len = sum(1 for _ in reader)

Moreover, we shouldnt forget pythonic code not always have the best performance in the project. In example: If we can do more operations at the same time in the same data set Its better to do all in the same bucle instead make two or more pythonic bucles.

1
  • 1
    Certainly a fastest solution. I'd recommend renaming the len variable as it's overwriting the built-in function.
    – s3dev
    Jan 21, 2021 at 14:17
3
numline = len(file_read.readlines())
2
  • 2
    file_read is apparently a csv.reader() object, so it does not have a readlines() method. .readlines() has to create a potentially large list, which you then discard again.
    – Martijn Pieters
    Apr 19, 2013 at 15:54
  • 1
    When i write this answer, topic haven't information about csv is csv reader object. Apr 19, 2013 at 16:09
3
import csv
count = 0
with open('filename.csv', 'rb') as count_file:
    csv_reader = csv.reader(count_file)
    for row in csv_reader:
        count += 1

print count
3

You can also use a classic for loop:

import pandas as pd
df = pd.read_csv('your_file.csv')

count = 0
for i in df['a_column']:
    count = count + 1

print(count)
1
  • 3
    If you're reading it as a DataFrame you don't need a loop you can just do len(df)
    – pyjamas
    Feb 16, 2021 at 7:21
2

Use "list" to fit a more workably object.

You can then count, skip, mutate till your heart's desire:

list(fileObject) #list values

len(list(fileObject)) # get length of file lines

list(fileObject)[10:] # skip first 10 lines
2

might want to try something as simple as below in the command line:

sed -n '$=' filename

or

wc -l filename
1
  • 1
    What if you have line breaks inside double quotes? That should still be considered part of the same record. This answer is wrong Nov 29, 2018 at 22:21
1
import pandas as pd
data = pd.read_csv('data.csv') 
totalInstances=len(data)
0

If you have to parse the CSV (e.g., because of the presence of line breaks in the fields or commented out lines) but the CSV is too large to fit the memory all at once, you might parse the CSV piece-by-piece:

import pandas as pd
import os
import sys

csv.field_size_limit(sys.maxsize)  # increase the maximal line length in pd.read_csv()

cnt = 0
for chunk in pd.read_csv(filepath, chunksize=10**6):
    cnt += len(chunk)
print(cnt)
0

I think mine will be the simplest approach here:

import csv
file = open(filename, 'r')
csvfile = csv.reader(file)
file.close
print("row", len(list(csvfile)))
1
  • This doesn't work if you do len(list(csvfile)) followed by "for index, row in enumerate(csvfile):", the enumerate() doesn't return any entries.
    – Samuel
    Mar 1, 2023 at 20:26
0

With pyarrow lib, is almost 6 times faster than dixhom suggested method.

👉 Used: csv with 3,921,865 rows and 927MB file size

Standard

sum(1 for _ in open(file_path))
# result: 3.57 s ± 90.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

With pyarrow

import pyarrow.csv as csv

sum([len(chunk) for chunk in csv.open_csv(file_path)])
# result: 854 ms ± 4.88 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
0
import pandas as pd
import csv

filename = 'data.csv'

row_count = sum(1 for line in open(filename))

# count no of lines 
print("Number of records : - ",row_count)

The result was : Number of records : - 163210690

-1

If you are working on a Unix system, the fastest method is the following shell command

cat FILE_NAME.CSV | wc -l

From Jupyter Notebook or iPython, you can use it with a !:

! cat FILE_NAME.CSV | wc -l
1
  • Why use 2 commands? wc -l FILE_NAME.CSV works just fine. May 25, 2023 at 15:42
-2

try

data = pd.read_csv("data.csv")
data.shape

and in the output you can see something like (aa,bb) where aa is the # of rows

3
  • Just stumbling across stuff, seems this shape comment isn't so bad and actually comparatively very fast: stackoverflow.com/questions/15943769/…
    – dedricF
    Mar 6, 2020 at 4:19
  • 2
    Oh but you'll want to do a data.shape[0]
    – dedricF
    Mar 6, 2020 at 4:20
  • But is it comparatively fast compared to @martijnpieters's answer, which uses a standard file handle/iterator, and doesn't require installing and importing the pandas library?
    – dancow
    Aug 19, 2020 at 1:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.