20

Is there a reasonably easy way to get data from some url? I tried the most obvious version, does not work:

readcsv("https://dl.dropboxusercontent.com/u/.../testdata.csv")

I did not find any usable reference. Any help?

6 Answers 6

23

If you want to read a CSV from a URL, you can use the Requests package as @waTeim shows and then read the data through an IOBuffer. See example below.

Or, as @Colin T Bowers comments, you could use the currently (December 2017) more actively maintained HTTP.jl package like this:

julia> using HTTP

julia> res = HTTP.get("https://www.ferc.gov/docs-filing/eqr/q2-2013/soft-tools/sample-csv/transaction.txt");

julia> mycsv = readcsv(res.body);

julia> for (colnum, myheader) in enumerate(mycsv[1,:])
           println(colnum, '\t', myheader)
       end
1   transaction_unique_identifier
2   seller_company_name
3   customer_company_name
4   customer_duns_number
5   tariff_reference
6   contract_service_agreement
7   trans_id
8   transaction_begin_date
9   transaction_end_date
10  time_zone
11  point_of_delivery_control_area
12  specific location
13  class_name
14  term_name
15  increment_name
16  increment_peaking_name
17  product_name
18  transaction_quantity
19  price
20  units
21  total_transmission_charge
22  transaction_charge

Using the Requests.jl package:

julia> using Requests

julia> res = get("https://www.ferc.gov/docs-filing/eqr/q2-2013/soft-tools/sample-csv/transaction.txt");

julia> mycsv = readcsv(IOBuffer(res.data));

julia> for (colnum, myheader) in enumerate(mycsv[1,:])
         println(colnum, '\t', myheader)
       end
1   transaction_unique_identifier
2   seller_company_name
3   customer_company_name
4   customer_duns_number
5   tariff_reference
6   contract_service_agreement
7   trans_id
8   transaction_begin_date
9   transaction_end_date
10  time_zone
11  point_of_delivery_control_area
12  specific location
13  class_name
14  term_name
15  increment_name
16  increment_peaking_name
17  product_name
18  transaction_quantity
19  price
20  units
21  total_transmission_charge
22  transaction_charge
1
  • 2017 update: HTTP.jl appears to be more actively maintained these days. Dec 12, 2017 at 5:05
16

If you are looking to read into a dataframe, this will also work in Julia:

using CSV   

dataset = CSV.read(download("https://mywebsite.edu/ml/machine-learning-databases/my.data"))
2
  • This is the most straightforward answer, gives a DataFrame output, seamless - thanks
    – Arnaud A
    Aug 30, 2020 at 19:48
  • Note that download is now under Downloads.download instead of Base Jul 23 at 16:23
10

The Requests package seems to work pretty well. There are others (see the entire package list) but Requests is actively maintained.

Obtaining it

julia> Pkg.add("Requests")

julia> using Requests

Using it

You can use one of the exported functions that correspond to the various HTTP verbs get, post, etc which returns a Response type

julia> res = get("http://julialang.org")
Response(200 OK, 21 Headers, 20913 Bytes in Body)

julia> typeof(res)
Response (constructor with 8 methods)

And then, for example, you can print the data using @printf

julia> @printf("%s",res.data);
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-us" lang="en-us">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
...
1
  • Alright, this is rather cumbersome. Shouldn't the question be converted into feature request? Or is there some benefit in keeping the infrastructure that complicated? Also couldn't the two replies from you and @rickhg12hs be merged, they complete each other very well.
    – krhlk
    Jun 16, 2014 at 7:32
3

If it is directly a csv file, something like this should work:

A = readdlm(download(url),';')
1
  • 1
    It is good to know that as said in the documentation: "Note that this function relies on the availability of external tools such as curl, wget or fetch to download the file and is provided for convenience."
    – ands
    Dec 30, 2018 at 23:40
2

Nowadays you can also use UrlDownload.jl which is pure Julia, take care of download details, process data in-memory and can also work with compressed files.

Usage is straightforward

using UrlDownload

A = urldownload("https://data.ok.gov/sites/default/files/unspsc%20codes_3.csv")
2

A very easy solution, alike to mike gold´s post, though in 2023 you need to specify a sink argument:

using CSV, DataFrames

my_table = CSV.read(download(some_url), DataFrame)
1
  • or kepping CSV and DataFrames "separately": CSV.download(some_url) |> CSV.File |> DataFrame Aug 28 at 22:39

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.