3

I am having issues reading a .dat file into a dataframe. I think the issue is with the delimiter. I have included a screen shot of what the data in the file looks like below. My best guess is that it is tab delimited between columns and then new-line delimited between rows. I have tried reading in the data with the following commands:

    df = CSV.File("FORCECHAIN00046.dat"; header=false) |> DataFrame!
    df = CSV.File("FORCECHAIN00046.dat"; header=false, delim = ' ') |> DataFrame!

My result either way is just a DataFrame with only one column including all the data frome each column concatenated into one string. I tried to even specify the types with the following code:

df = CSV.File("FORCECHAIN00046.dat"; types=[Float64,Float64,Float64,Float64,
Float64,Float64,Float64,Float64,Float64,Float64,Float64,Float64]) |> DataFrame!

And I received an the following error:

┌ Warning: 2; something went wrong trying to determine row positions for multithreading; it'd be very helpful if you could open an issue at https://github.com/JuliaData/CS
V.jl/issues so package authors can investigate

I can work around this by uploading it into google sheets and then downloading a csv, but I would like to find a way to make the original .dat file work.

enter image description here

3
  • 1
    have you tried delim=' ', ignorerepeated=true? Commented May 7, 2020 at 20:11
  • might not work because of the leading spaces that seem to be in there too Commented May 7, 2020 at 21:22
  • leading and trailing whitespace should not be a problem if ignorerepeated=true Commented May 7, 2020 at 21:38

2 Answers 2

4

Part of the issue here is that .dat is not a proper file format—it's just something that seems to be written out in a somewhat human-readable format with columns of numbers separated by variable numbers of spaces so that the numbers line up when you look at them in an editor. Google Sheets has a lot of clever tricks built in to "do what you want" for all kinds of ill-defined data files, so I'm not too surprised that it manages to parse this. The CSV package on the other hand supports using a single character as a delimiter or even a multi-character string, but not a variable number of spaces like this.

Possible solutions:

  • if the files aren't too big, you could easily roll your own parser that splits each line and then builds a matrix
  • you can also pre-process the file turning multiple spaces into single spaces

That's probably the easiest way to do this and here's some Julia code (untested since you didn't provide test data) that will open your file and convert it to a more reasonable format:

function dat2csv(dat_path::AbstractString, csv_path::AbstractString)
    open(csv_path, write=true) do io
        for line in eachline(dat_path)
            join(io, split(line), ',')
            println(io)
        end
    end
    return csv_path
end

function dat2csv(dat_path::AbstractString)
    base, ext = splitext(dat_path)
    ext == ".dat" ||
        throw(ArgumentError("file name doesn't end with `.dat`"))
    return dat2csv(dat_path, "$base.csv")
end

You would call this function as dat2csv("FORCECHAIN00046.dat") and it would create the file FORCECHAIN00046.csv, which would be a proper CSV file using commas as delimiters. That won't work well if the files contain any values with commas in them, but it looks like they are just numbers, in which case it should be fine. So you can use this function to convert the files to proper CSV and then load that file with the CSV package.

A little explanation of the code:

  • the two-argument dat2csv method opens csv_path for writing and then calls eachline on dat_path to read one line form it at a time
  • eachline strips any trailing newline from each line, so each line will be bunch of numbers separated by whitespace with some leading and/or trailing whitespace
  • split(line) does the default splitting of line which splits it on whitespace, dropping any empty values—this leaves just the non-whitespace entries as strings in an array
  • join(io, split(line), ',') joins the strings in the array together, separated by the , character and writes that to the io write handle for csv_path
  • println(io) writes a newline after that—otherwise everything would just end up on a single very long line
  • the one-argument dat2csv method calls splitext to split the file name into a base name and an extension, checking that the extension is the expected .dat and calling the two-argument version with the .dat replaced by .csv
Sign up to request clarification or add additional context in comments.

2 Comments

wow, thank you for such a through and helpful response! It worked perfectly, but just a few questions on how it works. I understand the code to be joining each newline together with a comma, so how does it end up joining each column with a comma? Also what is the purpose of the println(io). Does it function like a write statement in this case?
I'll add some explanation in the answer rather than answer here.
0

Try using the readdlm function in DelimitedFiles library, and convert to DataFrame afterwards:

using DelimitedFiles, DataFrames

df = DataFrame(readdlm("FORCECHAIN00046.dat"), :auto)

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.