6

I have noticed that loading a CSV file using CSV.read is quite slow. For reference, I am attaching one example of time benchmark:

using CSV, DataFrames
file = download("https://github.com/foursquare/twofishes")
@time CSV.read(file, DataFrame)

Output: 
9.450861 seconds (22.77 M allocations: 960.541 MiB, 5.48% gc time)
297 rows × 2 columns

This is a random dataset, and a python alternate of such operation compiles in fraction of time compared to Julia. Since, julia is faster than python why is this operation takes this much time? Moreover, is there any faster alternate to reduce the compile timing?

3
  • I assume this is Julia 1.5? Jan 11, 2021 at 1:24
  • @OscarSmith yes i am using Julia 1.5.3 Jan 11, 2021 at 2:04
  • I don't think the similar operation in python does any compilation. Talking about faster compilation in python then may be inexact. Jan 11, 2021 at 18:26

1 Answer 1

7

You are measuring the compile together with runtime.

One correct way to measure the time would be:

@time CSV.read(file, DataFrame)
@time CSV.read(file, DataFrame)

At the first run the function compiles at the second run you can use it.

Another option is using BenchmarkTools:

using BenchmarkTools
@btime CSV.read(file, DataFrame)

Normally, one uses Julia to work with huge datasets so that single initial compile time is not important. However, it is possible to compile CSV and DataFrame into Julia's system image and have fast execution from the first run, for isntructions see here: Why julia takes long time to import a package? (this is however more advanced usually one does not need it)

You also have yet another option which is reducing the optimization level for the compiler (this would be for scenarios where your workload is small and restarted frequently and you do not want all complexity that comes with image building. In this cage you would run Julia as:

julia --optimize=0 my_code.jl

Finally, like mentioned by @Oscar Smith in the forthcoming Julia 1.6 the compile times will be slightly shorter.

8
  • 4
    I mostly disagree with this answer. For data analysis workflows this compile time matters a lot. I think a better answer would be to show 1.6 and how it decreases the time taken Jan 11, 2021 at 1:28
  • You can compile this into system image or set the optimization level (just finished updating the anwser). And yes 1.6 is great! Jan 11, 2021 at 1:29
  • 3
    If this is a big production cluster with many short living processes I would go with building a custom Julia system image (if it is not possible to redesign the parameter sweep in such a way that a process stays alive for something like 15mins). Jan 11, 2021 at 2:17
  • 2
    Also check out github.com/dmolina/DaemonMode.jl Jan 11, 2021 at 5:16
  • 1
    An update: I used PackageCompiler package to create sysimage and it works but unfortunately, it doesn't do much for functions using PyCall however it improves the loading time for pure julia codes by 50%. Jan 14, 2021 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.