4

For a CSV file, I am using a combination of Pipe, HTTP and CSV to download, (eventually) modify the file and import it as a DataFrame without any temporary writing on disk:

urlData = "https://github.com/sylvaticus/IntroSPMLJuliaCourse/raw/main/lessonsSources/02_-_JULIA2_-_Scientific_programming_with_Julia/data.csv"
data = @pipe HTTP.get(urlData).body                |>
             replace!(_, UInt8(';') => UInt8(' ')) |>  # if we need to apply modifications to the file before importing 
             CSV.File(_, delim=' ')                |>
             DataFrame;

How can I use the same general approach (pipe without temp disk saving) when the file has been compressed with zip or tar (assuming a single file in the archive) ?

For example:

urlDataZ = "https://github.com/sylvaticus/IntroSPMLJuliaCourse/raw/main/lessonsSources/02_-_JULIA2_-_Scientific_programming_with_Julia/data.zip"
urlDataT = "https://github.com/sylvaticus/IntroSPMLJuliaCourse/raw/main/lessonsSources/02_-_JULIA2_-_Scientific_programming_with_Julia/data.tgz"

For zipped files I can use this approach:

data = @pipe ZipFile.Reader("data.zip").files[1] |>
             CSV.File(read(_), delim=';') |>
             DataFrame  

But I can't arrive to put the three things (download, decompress and import) together without passing for an explicit temp file on disk.

1 Answer 1

3

ZipFile.Reader accepts IO arguments too, so you can wrap things in an IOBuffer like this:

julia> @pipe HTTP.get(urlDataZ).body             |>
             IOBuffer(_)                         |>
             ZipFile.Reader(_).files[1]          |>
             CSV.read(_, DataFrame, delim = ';')
             
8×4 DataFrame
 Row │ Country  Year   forarea   forvol  
     │ String7  Int64  Float64   Float64 
─────┼───────────────────────────────────
   1 │ Germany   2000  11.354    3381.0
   2 │ France    2000  15.288    2254.28
   3 │ Italy     2000   8.36925  1058.71
   4 │ Sweden    2000  28.163    3184.67
   5 │ Germany   2020  11.419    3663.0
   6 │ France    2020  17.253    3055.83
   7 │ Italy     2020   9.56613  1424.4
   8 │ Sweden    2020  27.98     3653.91

Sign up to request clarification or add additional context in comments.

2 Comments

Thank you can you explain a bit more about the Tar.jl solution? It seems that Tar.extract() extract to a directory, I don't know how to extract to stream, and also after how to decompress, as it is g-zipped, not zipped. I did open a thread on discourse, but the solution I got there is not very concise.. but maybe there isn't better one...
(Using Tar without creating intermediate files appears more complicated, I'll update the answer when I figure that one out.) @Antonello Ha, I just added this comment before the page updated with yours. Yeah, the Tar handling isn't as straightforward as I assumed, but I still think it's possible. I'll add it in if/when I have it working.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.