7

I had a hard time using Julia to read a large text file (968MB, 8.7 million rows). Each line is like:

0.3295747E+01   0.3045123E+01   0.3325542E+01   0.1185458E+01  -0.4827727E-05  -0.1033694E-04   0.3306459E-03

I used parse.(Float64, split(line)) to convert every line to numbers.

function openfile()
    datafile = open("data.dat","r")
    lines = readlines(datafile)
    close(datafile)
    lines
end

function parseline(lines::Array{String})
    for line in lines
        zzz = parse.(Float64, split(line))
    end
end

import Base: tryparse_internal
function myreadfile(str::String, T::Type, dlm=' ', eol='\n')
    row, clm, bg, ed = 0, 0, 0, 0
    data = Array{T}(undef,0)
    isnu0, isnu = false, false
    for (idx, chr) in enumerate(str)
        isnu = false
        (chr != eol && chr != dlm) && (isnu = true)
        if isnu0 == false && isnu == true
            bg, isnu0 = idx, true
        end
        if isnu0 == true && isnu == false
            ed, isnu0 = idx-1, false
            push!(data, tryparse_internal(T, str, bg, ed))
        end
    end
    isnu == true && (push!(data, tryparse(T, str[bg:end])))
    data
end

@time lines = openfile()
@time parseline(lines)
using DelimitedFiles
@time readdlm("data.dat")
@time myreadfile(read("data.dat",String), Float64)

and got

  3.584656 seconds (17.59 M allocations: 1.240 GiB, 28.44% gc time)
 78.099010 seconds (276.14 M allocations: 6.080 GiB, 1.50% gc time)
 52.504199 seconds (185.93 M allocations: 3.960 GiB, 0.53% gc time)
 46.085581 seconds (61.70 M allocations: 2.311 GiB, 0.28% gc time)

Compare with fortran code

call cpu_time(start)
open(10, file="data.dat",status="old")
do i=1, 8773632
    read(10,*) a, b, c, d, e, f, g
end do
call cpu_time(finish)
print '("Time = ",f6.3," seconds.")',finish-start

Which is Time = 14.812 seconds.

It seems Julia spends much longer time doing the same thing. Is there a better way to convert string to float? split and parse are so slow.

3
  • Tried to use the readdlm function from DelimitedFiles package? Commented Mar 20, 2020 at 0:55
  • 1
    Is the Fortran code actually converting to float? Commented Mar 20, 2020 at 5:30
  • 1
    Of course it does Commented Mar 20, 2020 at 16:50

2 Answers 2

5

As the comment says above, the fastest is most likely the readdlm function. That will return a Matrix which is most likely what you want.

If you do want to do it by hand it's usually better to read through the file and process it line by line, instead of storing everything in big intermediary objects. Memory reads and writes are slow. Something like

ret = open("data.dat","r") do datafile
    [parse.(Float64, split(line)) for line in eachline(datafile)]
end

It's probably not faster than your last line anyway though.

Sign up to request clarification or add additional context in comments.

4 Comments

An alternative would be using CSV.jl which in particular can read in the source file using multiple threads.
"Memory reads and writes are slow in julia": why in julia? Julia is just using the hardware.
there are some options when using readdlm, try ?readdlm to see them.
If nothing is useful for now, maybe you can consider Calling C and Fortran Code
3

Though I don't know your Fortran compiler, I will hazard a guess here.

I think the difference in execution time is because the Julia code is doing two things your Fortran compiler does do not at the time the files are processed. The extra work Julia does makes Julia more versatile overall in its file handling than Fortran, but definitely slow it down here, since in this case they are not needed:

  1. Julia is using Unicode strings, 32 bits per digit rather than Fortran's 8 bits, which slows the assembly level byte comparisons in conversion of file to text line and splitting of text lines.

  2. I think Fortran is doing 32-bit floats and not 64-bit floating point numbers, whereas Julia is definitely using 64-bit floating point here, which might in some situations double the conversion times to floating point.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.