216

We have a large raw data file that we would like to trim to a specified size.

How would I go about getting the first N lines of a text file in python? Will the OS being used have any effect on the implementation?

1
  • can I give n as command line argument
    – user6882757
    Jul 23, 2019 at 9:07

20 Answers 20

334

Python 3:

with open(path_to_file) as input_file:
    head = [next(input_file) for _ in range(lines_number)]
print(head)

Python 2:

with open(path_to_file) as input_file:
    head = [next(input_file) for _ in xrange(lines_number)]
print head

Here's another way (both Python 2 & 3):

from itertools import islice

with open(path_to_file) as input_file:
    head = list(islice(input_file, lines_number))
print(head)
16
  • 1
    Thanks, that is very helpful indeed. What is the difference between the two? (in terms of performance, required libraries, compatibility etc)?
    – Russell
    Nov 20, 2009 at 0:34
  • 1
    I expect the performance to be similar, maybe the first to be slightly faster. But the first one won't work if the file doesn't have at least N lines. You are best to measure the performance against some typical data you will be using it with. Nov 20, 2009 at 0:47
  • 1
    The with statement works on Python 2.6, and requires an extra import statement on 2.5. For 2.4 or earlier, you'd need to rewrite the code with a try...except block. Stylistically, I prefer the first option, although as mentioned the second is more robust for short files.
    – Alasdair
    Nov 20, 2009 at 1:21
  • 1
    islice is probably faster as it is implemented in C. Nov 20, 2009 at 6:45
  • 35
    Have in mind that if the files have less then N lines this will raise StopIteration exception that you must handle Jan 25, 2012 at 12:44
31
N = 10
with open("file.txt", "a") as file:  # the a opens it in append mode
    for i in range(N):
        line = next(file).strip()
        print(line)
4
  • 4
    Why open the file in append mode?
    – AMC
    Apr 8, 2020 at 0:16
  • @AMC I think it is for not deleting the file, but we should use 'r' here instead. Aug 18, 2020 at 14:23
  • 1
    @Kowalski Append mode is for adding to the file, r is indeed the more logical choice, I think.
    – AMC
    Aug 19, 2020 at 18:35
  • @ghostdog74, how can read the next N values ?
    – lena
    Dec 7, 2020 at 10:08
28

If you want to read the first lines quickly and you don't care about performance you can use .readlines() which returns list object and then slice the list.

E.g. for the first 5 lines:

with open("pathofmyfileandfileandname") as myfile:
    firstNlines=myfile.readlines()[0:5] #put here the interval you want

Note: the whole file is read so is not the best from the performance point of view but it is easy to use, fast to write and easy to remember so if you want just perform some one-time calculation is very convenient

print firstNlines

One advantage compared to the other answers is the possibility to select easily the range of lines e.g. skipping the first 10 lines [10:30] or the lasts 10 [:-10] or taking only even lines [::2].

7
  • 3
    The top answer is probably way more efficient, but this one works like a charm for small files. Nov 7, 2015 at 12:53
  • 3
    Note that this actually reads the whole file into a list first (myfile.readlines()) and then splices the first 5 lines of it. Oct 25, 2016 at 9:07
  • 6
    This should be avoided.
    – anilbey
    Nov 27, 2018 at 22:58
  • 1
    I see no reason to use this, it's not any simpler than the vastly more efficient solutions.
    – AMC
    Apr 8, 2020 at 0:17
  • 1
    @AMC thanks for the feedback, I use it in the console for exploring the data when I have to have a quick look to the first lines, it just saves me time in writing code.
    – G M
    Apr 8, 2020 at 8:27
13

What I do is to call the N lines using pandas. I think the performance is not the best, but for example if N=1000:

import pandas as pd
yourfile = pd.read_csv('path/to/your/file.csv',nrows=1000)
4
  • 4
    Better would be to use the nrows option, which can be set to 1000 and the entire file isn't loaded. pandas.pydata.org/pandas-docs/stable/generated/… In general, pandas has this and other memory-saving techniques for big files.
    – philshem
    Apr 11, 2017 at 15:03
  • Yes, you are right. I just correct it. Sorry for the mistake.
    – RRuiz
    Apr 11, 2017 at 15:06
  • 1
    You may also want to add sep to define a column delimiter (which shouldn't occur in a non-csv file)
    – philshem
    Apr 11, 2017 at 15:09
  • 2
    @Cro-Magnon I cannot find the pandas.read() function in the documentation, do you know of any information on the subject?
    – AMC
    Apr 8, 2020 at 0:19
8

There is no specific method to read number of lines exposed by file object.

I guess the easiest way would be following:

lines =[]
with open(file_name) as f:
    lines.extend(f.readline() for i in xrange(N))
1
  • This is something I had actually intended. Though, I though of adding each line to list. Thank you.
    – artdanil
    Nov 20, 2009 at 2:11
6

The two most intuitive ways of doing this would be:

  1. Iterate on the file line-by-line, and break after N lines.

  2. Iterate on the file line-by-line using the next() method N times. (This is essentially just a different syntax for what the top answer does.)

Here is the code:

# Method 1:
with open("fileName", "r") as f:
    counter = 0
    for line in f:
        print line
        counter += 1
        if counter == N: break

# Method 2:
with open("fileName", "r") as f:
    for i in xrange(N):
        line = f.next()
        print line

The bottom line is, as long as you don't use readlines() or enumerateing the whole file into memory, you have plenty of options.

1
  • The bottom line is, as long as you don't use readlines() or enumerateing the whole file into memory, you have plenty of options. Isn't enumerate() lazy?
    – AMC
    Aug 19, 2020 at 18:40
4

Based on gnibbler top voted answer (Nov 20 '09 at 0:27): this class add head() and tail() method to file object.

class File(file):
    def head(self, lines_2find=1):
        self.seek(0)                            #Rewind file
        return [self.next() for x in xrange(lines_2find)]

    def tail(self, lines_2find=1):  
        self.seek(0, 2)                         #go to end of file
        bytes_in_file = self.tell()             
        lines_found, total_bytes_scanned = 0, 0
        while (lines_2find+1 > lines_found and
               bytes_in_file > total_bytes_scanned): 
            byte_block = min(1024, bytes_in_file-total_bytes_scanned)
            self.seek(-(byte_block+total_bytes_scanned), 2)
            total_bytes_scanned += byte_block
            lines_found += self.read(1024).count('\n')
        self.seek(-total_bytes_scanned, 2)
        line_list = list(self.readlines())
        return line_list[-lines_2find:]

Usage:

f = File('path/to/file', 'r')
f.head(3)
f.tail(3)
3

most convinient way on my own:

LINE_COUNT = 3
print [s for (i, s) in enumerate(open('test.txt')) if i < LINE_COUNT]

Solution based on List Comprehension The function open() supports an iteration interface. The enumerate() covers open() and return tuples (index, item), then we check that we're inside an accepted range (if i < LINE_COUNT) and then simply print the result.

Enjoy the Python. ;)

1
  • This just seems like a slightly more complex alternative to [next(file) for _ in range(LINE_COUNT)].
    – AMC
    Apr 8, 2020 at 0:20
3

For first 5 lines, simply do:

N=5
with open("data_file", "r") as file:
    for i in range(N):
       print file.next()
2

If you want something that obviously (without looking up esoteric stuff in manuals) works without imports and try/except and works on a fair range of Python 2.x versions (2.2 to 2.6):

def headn(file_name, n):
    """Like *x head -N command"""
    result = []
    nlines = 0
    assert n >= 1
    for line in open(file_name):
        result.append(line)
        nlines += 1
        if nlines >= n:
            break
    return result

if __name__ == "__main__":
    import sys
    rval = headn(sys.argv[1], int(sys.argv[2]))
    print rval
    print len(rval)
0
2

If you have a really big file, and assuming you want the output to be a numpy array, using np.genfromtxt will freeze your computer. This is so much better in my experience:

def load_big_file(fname,maxrows):
'''only works for well-formed text file of space-separated doubles'''

rows = []  # unknown number of lines, so use list

with open(fname) as f:
    j=0        
    for line in f:
        if j==maxrows:
            break
        else:
            line = [float(s) for s in line.split()]
            rows.append(np.array(line, dtype = np.double))
            j+=1
return np.vstack(rows)  # convert list of vectors to array
1
  • If you have a really big file, and assuming you want the output to be a numpy array That's quite a unique set of restrictions, I can't really see any advantages to this over the alternatives.
    – AMC
    Apr 8, 2020 at 0:22
1

This worked for me

f = open("history_export.csv", "r")
line= 5
for x in range(line):
    a = f.readline()
    print(a)
1
  • Why not use a context manager? In any case, I don't see how this improves on the many existing answers.
    – AMC
    Apr 8, 2020 at 0:24
1

I would like to handle the file with less than n-lines by reading the whole file

def head(filename: str, n: int):
    try:
        with open(filename) as f:
            head_lines = [next(f).rstrip() for x in range(n)]
    except StopIteration:
        with open(filename) as f:
            head_lines = f.read().splitlines()
    return head_lines

Credit go to John La Rooy and Ilian Iliev. Use the function for the best performance with exception handle

Revise 1: Thanks FrankM for the feedback, to handle file existence and read permission we can futher add

import errno
import os

def head(filename: str, n: int):
    if not os.path.isfile(filename):
        raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filename)  
    if not os.access(filename, os.R_OK):
        raise PermissionError(errno.EACCES, os.strerror(errno.EACCES), filename)     
   
    try:
        with open(filename) as f:
            head_lines = [next(f).rstrip() for x in range(n)]
    except StopIteration:
        with open(filename) as f:
            head_lines = f.read().splitlines()
    return head_lines

You can either go with second version or go with the first one and handle the file exception later. The check is quick and mostly free from performance standpoint

2
  • Well this isn't soundproof. Meaning if there is an exception, you try to read the file again, which could throw another exception. This works if the file exists and you got the permissions to read. If not it results in an exception. The accepted answer provides (solution 3) a variant which does the same using islice (reads the whole file, when it has fewer lines). But your solution is better than variant 1 and 2.
    – FrankM
    Jul 9, 2021 at 10:09
  • Thanks @FrankM for the feedback, please see my revise answer
    – Linh K Ha
    Jul 10, 2021 at 13:06
0

Starting at Python 2.6, you can take advantage of more sophisticated functions in the IO base clase. So the top rated answer above can be rewritten as:

    with open("datafile") as myfile:
       head = myfile.readlines(N)
    print head

(You don't have to worry about your file having less than N lines since no StopIteration exception is thrown.)

3
  • 27
    According to the docs N is the number of bytes to read, not the number of lines. Jun 18, 2013 at 17:41
  • 4
    N is the number of bytes!
    – qed
    Jun 1, 2014 at 14:19
  • 6
    Wow. Talk about poor naming. The function name mentions lines but the argument refers to bytes. Apr 27, 2015 at 18:22
0

This works for Python 2 & 3:

from itertools import islice

with open('/tmp/filename.txt') as inf:
    for line in islice(inf, N, N+M):
        print(line)
1
0

fname = input("Enter file name: ")
num_lines = 0

with open(fname, 'r') as f: #lines count
    for line in f:
        num_lines += 1

num_lines_input = int (input("Enter line numbers: "))

if num_lines_input <= num_lines:
    f = open(fname, "r")
    for x in range(num_lines_input):
        a = f.readline()
        print(a)

else:
    f = open(fname, "r")
    for x in range(num_lines_input):
        a = f.readline()
        print(a)
        print("Don't have", num_lines_input, " lines print as much as you can")


print("Total lines in the text",num_lines)

0

Here's another decent solution with a list comprehension:

file = open('file.txt', 'r')

lines = [next(file) for x in range(3)]  # first 3 lines will be in this list

file.close()
1
  • 1
    How is this answer different from the accepted answer?!
    – zardosht
    Mar 15, 2022 at 10:52
0

An easy way to get first 10 lines:

with open('fileName.txt', mode = 'r') as file:
    list = [line.rstrip('\n') for line in file][:10]
    print(list)
-2
#!/usr/bin/python

import subprocess

p = subprocess.Popen(["tail", "-n 3", "passlist"], stdout=subprocess.PIPE)

output, err = p.communicate()

print  output

This Method Worked for me

2
  • This isn't really a Python solution, though.
    – AMC
    Apr 8, 2020 at 0:23
  • I do not even understand what is written in your answer. Please add some explanation.
    – Nairum
    Jun 30, 2020 at 13:09
-2

Simply Convert your CSV file object to a list using list(file_data)

import csv;
with open('your_csv_file.csv') as file_obj:
    file_data = csv.reader(file_obj);
    file_list = list(file_data)
    for row in file_list[:4]:
        print(row)
1
  • Will be horribly slow for huge files, since you'll have to load every single line just to get first 4 of them Nov 20, 2021 at 14:52

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.