565

Is there a direct way to import the contents of a CSV file into a record array, just like how R's read.table(), read.delim(), and read.csv() import data into R dataframes?

Or should I use csv.reader() and then apply numpy.core.records.fromrecords()?

0

14 Answers 14

866

Use numpy.genfromtxt() by setting the delimiter kwarg to a comma:

from numpy import genfromtxt
my_data = genfromtxt('my_file.csv', delimiter=',')
8
  • 17
    What if you want something of different types? Like strings and ints? Mar 21, 2017 at 2:20
  • 15
    @CGTheLegend np.genfromtxt('myfile.csv',delimiter=',',dtype=None) Apr 26, 2017 at 2:45
  • 5
    numpy.loadtxt worked pretty well for me too
    – Yibo Yang
    May 19, 2017 at 17:34
  • 12
    I tried this but I am only getting nan values, why? Also with loadtxt, I am getting UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 155: ordinal not in range(128). I have umlauts such as ä and ö in the input data.
    – hhh
    Jun 18, 2017 at 12:00
  • 5
    @hhh try adding encoding="utf8" argument. Python is one of the few modern software pieces that frequently causes text encoding problems, which feel as things from the past.
    – kolen
    Sep 24, 2018 at 22:34
241

Use pandas.read_csv:

import pandas as pd
df = pd.read_csv('myfile.csv', sep=',', header=None)
print(df.values)
array([[ 1. ,  2. ,  3. ],
       [ 4. ,  5.5,  6. ]])

This gives a pandas DataFrame which provides many useful data manipulation functions which are not directly available with numpy record arrays.

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table...


I would also recommend numpy.genfromtxt. However, since the question asks for a record array, as opposed to a normal array, the dtype=None parameter needs to be added to the genfromtxt call:

import numpy as np
np.genfromtxt('myfile.csv', delimiter=',')

For the following 'myfile.csv':

1.0, 2, 3
4, 5.5, 6

the code above gives an array:

array([[ 1. ,  2. ,  3. ],
       [ 4. ,  5.5,  6. ]])

and

np.genfromtxt('myfile.csv', delimiter=',', dtype=None)

gives a record array:

array([(1.0, 2.0, 3), (4.0, 5.5, 6)], 
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<i4')])

This has the advantage that files with multiple data types (including strings) can be easily imported.

5
  • 2
    read_csv works with commas inside quotes. Recommend this over genfromtxt
    – Viet
    Apr 6, 2016 at 21:37
  • 3
    use header=0 to skip the first line in the values, if your file has a 1-line header
    – c-chavez
    Jun 30, 2017 at 13:34
  • Bear in mind that this creates a 2d array: e.g. (1000, 1). np.genfromtxt does not do that: e.g. (1000,).
    – Newskooler
    May 12, 2020 at 18:38
  • 1
    The OP is asking for Numpy arrays, not about Pandas Dataframe objects. Apr 6 at 13:43
  • @JoséL.Patiño The second part of the question deals with the request for a Numpy record array. The first part of the answer shows df.values which gives a Numpy representation of the DataFrame; a convenient method imho.
    – Lee
    Apr 6 at 17:36
93

I tried it :

from numpy import genfromtxt
genfromtxt(fname = dest_file, dtype = (<whatever options>))

versus :

import csv
import numpy as np
with open(dest_file,'r') as dest_f:
    data_iter = csv.reader(dest_f,
                           delimiter = delimiter,
                           quotechar = '"')
    data = [data for data in data_iter]
data_array = np.asarray(data, dtype = <whatever options>)

on 4.6 million rows with about 70 columns and found that the NumPy path took 2 min 16 secs and the csv-list comprehension method took 13 seconds.

I would recommend the csv-list comprehension method as it is most likely relies on pre-compiled libraries and not the interpreter as much as NumPy. I suspect the pandas method would have similar interpreter overhead.

2
  • 30
    I tested code similar to this with a csv file containing 2.6 million rows and 8 columns. numpy.recfromcsv() took about 45 seconds, np.asarray(list(csv.reader())) took about 7 seconds, and pandas.read_csv() took about 2 seconds (!). (The file had recently been read from disk in all cases, so it was already in the operating system's file cache.) I think I'll go with pandas. Mar 31, 2016 at 21:56
  • 6
    I just noticed there are some notes about the design of pandas' fast csv parser at wesmckinney.com/blog/… . The author takes speed and memory requirements pretty seriously. It's also possible to use as_recarray=True to get the result directly as a Python record array rather than a pandas dataframe. Apr 5, 2016 at 19:20
70

You can also try recfromcsv() which can guess data types and return a properly formatted record array.

1
  • 11
    If you want to maintain ordering / column names in the CSV, you can use the following invocation: numpy.recfromcsv(fname, delimiter=',', filling_values=numpy.nan, case_sensitive=True, deletechars='', replace_space=' ') The key arguments are the last three. Oct 17, 2013 at 14:00
26

As I tried both ways using NumPy and Pandas, using pandas has a lot of advantages:

  • Faster
  • Less CPU usage
  • 1/3 RAM usage compared to NumPy genfromtxt

This is my test code:

$ for f in test_pandas.py test_numpy_csv.py ; do  /usr/bin/time python $f; done
2.94user 0.41system 0:03.05elapsed 109%CPU (0avgtext+0avgdata 502068maxresident)k
0inputs+24outputs (0major+107147minor)pagefaults 0swaps

23.29user 0.72system 0:23.72elapsed 101%CPU (0avgtext+0avgdata 1680888maxresident)k
0inputs+0outputs (0major+416145minor)pagefaults 0swaps

test_numpy_csv.py

from numpy import genfromtxt
train = genfromtxt('/home/hvn/me/notebook/train.csv', delimiter=',')

test_pandas.py

from pandas import read_csv
df = read_csv('/home/hvn/me/notebook/train.csv')

Data file:

du -h ~/me/notebook/train.csv
 59M    /home/hvn/me/notebook/train.csv

With NumPy and pandas at versions:

$ pip freeze | egrep -i 'pandas|numpy'
numpy==1.13.3
pandas==0.20.2
9

Using numpy.loadtxt

A quite simple method. But it requires all the elements being float (int and so on)

import numpy as np 
data = np.loadtxt('c:\\1.csv',delimiter=',',skiprows=0)  
1
  • 1
    Also can use this: ''' data2 = np.genfromtxt(''c:\\1.csv', delimiter=',') ''' Oct 21, 2021 at 19:53
7

You can use this code to send CSV file data into an array:

import numpy as np
csv = np.genfromtxt('test.csv', delimiter=",")
print(csv)
7

This work as a charm...

import csv
with open("data.csv", 'r') as f:
    data = list(csv.reader(f, delimiter=";"))

import numpy as np
data = np.array(data, dtype=np.float)
0
6

I would suggest using tables (pip3 install tables). You can save your .csv file to .h5 using pandas (pip3 install pandas),

import pandas as pd
data = pd.read_csv("dataset.csv")
store = pd.HDFStore('dataset.h5')
store['mydata'] = data
store.close()

You can then easily, and with less time even for huge amount of data, load your data in a NumPy array.

import pandas as pd
store = pd.HDFStore('dataset.h5')
data = store['mydata']
store.close()

# Data in NumPy format
data = data.values
6

This is the easiest way:

import csv
with open('testfile.csv', newline='') as csvfile:
    data = list(csv.reader(csvfile))

Now each entry in data is a record, represented as an array. So you have a 2D array. It saved me so much time.

1
  • 3
    Why should we have to screw around with Pandas, when these tools have so much less feature bloat?
    – Chris
    Jan 7, 2020 at 0:50
6

Available on the newest pandas and numpy version.

import pandas as pd
import numpy as np

data = pd.read_csv('data.csv', header=None)

# Discover, visualize, and preprocess data using pandas if needed.

data = data.to_numpy()
4

I tried this:

import pandas as p
import numpy as n

closingValue = p.read_csv("<FILENAME>", usecols=[4], dtype=float)
print(closingValue)
0
In [329]: %time my_data = genfromtxt('one.csv', delimiter=',')
CPU times: user 19.8 s, sys: 4.58 s, total: 24.4 s
Wall time: 24.4 s

In [330]: %time df = pd.read_csv("one.csv", skiprows=20)
CPU times: user 1.06 s, sys: 312 ms, total: 1.38 s
Wall time: 1.38 s
1
  • 1
    Please edit the question with some more information about your solution.
    – Ruli
    Jan 13, 2021 at 8:31
-1

this is a very simple task, the best way to do this is as follows

import pandas as pd
import numpy as np


df = pd.read_csv(r'C:\Users\Ron\Desktop\Clients.csv')   #read the file (put 'r' before the path string to address any special characters in the file such as \). Don't forget to put the file name at the end of the path + ".csv"

print(df)`

y = np.array(df)
2
  • 2
    The OP asked to read directly to numpy array. Reading it as a dataframe and converting it to numpy array requires more storage and time. Aug 3, 2022 at 21:21
  • Yes, that's correct. But I just gave another possible way of doing the same thing, if the above doesn't work
    – Ovu Sunday
    Aug 4, 2022 at 11:23

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.