7

I just started learning Julia and I want to read many csv files in my directory. How can I do that?

My directory has the files below and I want to read in all files from trip_data_1 to trip_data_12.

"trip_data_1.csv" "trip_data_10.csv" "trip_data_11.csv" "trip_data_12.csv" "trip_data_2.csv" "trip_data_3.csv" "trip_data_4.csv" "trip_data_5.csv" "trip_data_6.csv" "trip_data_7.csv" "trip_data_8.csv" "trip_data_9.csv" "trip_fare_1.csv" "trip_fare_10.csv" "trip_fare_11.csv" "trip_fare_12.csv" "trip_fare_2.csv" "trip_fare_3.csv" "trip_fare_4.csv" "trip_fare_5.csv" "trip_fare_6.csv" "trip_fare_7.csv" "trip_fare_8.csv" "trip_fare_9.csv"

This is what I have tried:

using DataFrames
df = readtable(filter!(r"^trip_data", readdir()))

But I get MethodError: no method matching readtable(::Array{String,1})

2
  • 1
    filter! (in this scenario) returns Array{String, 1}, i.e. a vector of all your file-names. But readtable can only read one file at a time, and so needs a String input. Solution? Just iterate over the output of your filter! operation with calls to readtable. Mar 1, 2017 at 4:03
  • Thanks! That helps! Mar 1, 2017 at 14:15

6 Answers 6

6

I'm a big fan of . broadcasting syntax in this type of situation.

I.e. df = readtable.(filter(r"^trip_data", readdir())) will give you an array of data frames (@avysk is correct that you probably want filter not filter!.

If you want one single data frame then the mapreduceoption is good.

Or you can: vcat(readtable.(filter(r"^trip_data", readdir()))

NB: All of these are general solutions to the problem, I have a function (method) that applies f to x and now I want to apply it to many instances, or an array, of x

So if you get another error that indicates that you cannot apply a function directly to any array or collection, but you can to a single element, then map, broadcast/. & list comprehensions are your friends!

2
  • Nice approach with broadcasting! However, in case of "I need one dataframe" I think mapreduce (not reduce(... map(...)), though) is the best, since it doesn't duplicate the data in memory -- all other approaches first create a sequence of dataframes, which is kept in memory until the creation of the final result is done. On the other hand, mapreduce doesn't build intermediate sequence. Or am I wrong?
    – avysk
    Mar 1, 2017 at 11:06
  • Nope, that's correct. The only reason I can think of to create an intermediate collection first is if you need to keep track of which file each row/frame originated from. Mar 1, 2017 at 11:59
5

You can do it like this:

reduce(vcat,  map(readtable, filter(r"^trip_data", readdir())))

Here map applies readtable to every filename matched by filter (you don't need filter! here) and joins all resulting dataframes together (vcat).

The same can be written with mapreduce:

mapreduce(readtable, vcat, filter(r"^trip_data", readdir()))
4

Here's a solution using Glob that I believe is more readable:

using CSV, Glob, DataFrames

files = glob("trip_data_*.csv")
dfs = CSV.read.(files)

As mentioned in another answer, CSV.read. (with the final dot) broadcasts over the files array, and creates an array of DataFrames. Therefore, the last line is equivalent to:

dfs = [CSV.read(file) for file in files]

Finally, if you want to concatenate all the files, you can simply do:

df = vcat(dfs...)

where the ... is used to pass a variable number of arguments to a function.

1
  • Hello. I find this solution very easy and useful. How can one choose a different file directory? Dec 6, 2023 at 9:58
3

Another method (which moves concatenation to the input String level instead of DataFrame level) and uses Iterators package:

readtable(IOBuffer(join(chain([drop((l for l in readlines(fn)),i>1?1:0) for (i,fn) in enumerate(filter!(r"^trip_data", readdir()))]...))))

This may actually save some time and allocations (in my pet example it did), but it depends on the parameters of the input files.

1

You could also do a simple

files = filter(r".csv$", readdir(path))
df = vcat([readtable(f) for f in files])

and as a follow up, I did the same with julia's CSV.read(file) and this is much slower. Actually not the reading part, but the sourcing part:

source = CSV.Source(file)
CSV.Read(source)
0

Supplementing the answers above. It seems that CSV.read will be deprecated. In addition it appears some users had problems deleting rows from DataFrames when using CSV.read directly. This code posted here using CSV.File might also help.

using CSV, Glob, DataFrame  
folder = "d:/Data/Test/" 
files = glob("*.csv", folder) 
df3=DataFrame.(CSV.File.(files))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.