25

I'm trying to analyze data stored in an SQL database (MS SQL server) in R, and on a mac. Typical queries might return a few GB of data, and the entire database is a few TB. So far, I've been using the R package odbc, and it seems to work pretty well.

However, dbFetch() seems really slow. For example, a somewhat complex query returns all results in ~6 minutes in SQL server, but if I run it with odbc and then try dbFetch, it takes close to an hour to get the full 4 GB into a data.frame. I've tried fetching in chunks, which helps modestly: https://stackoverflow.com/a/59220710/8400969. I'm wondering if there is another way to more quickly pipe the data to my mac, and I like the line of thinking here: Quickly reading very large tables as dataframes

What are some strategies for speeding up dbFetch when the results of queries are a few GB of data? If the issue is generating a data.frame object from larger tables, are there savings available by "fetching" in a different manner? Are there other packages that might help?

Thanks for your ideas and suggestions!

14
  • I was just reading in large tables with data.table::fread and read.csv. fread(my.csv, data.table=F) beats out fread(my.csv), and both take 1/10 as long as read.csv. I don't yet know much about how fread works but is there an equivalent method for fetching query results? Commented Mar 18, 2020 at 18:26
  • As I learn more about fread, wondering if there is a way to dbFetch by mmaping the query results for fast read in. Commented Mar 19, 2020 at 12:40
  • 1
    You might be better off running the query in SSMS with "Results to Text". That will create a csv flat file - try fread on that csv? Commented Mar 24, 2020 at 21:22
  • 1
    I found this helps in some situations: db.rstudio.com/pool
    – BSCowboy
    Commented Jan 8, 2021 at 23:07
  • 1
    I have been running into similar problems on my projects, and I found the solution usually to be to reduce the amount of data transfered. There's a reason large datatables reside on databases. The most user-friendly packages I've encounterd is dbplyr, which allows dplyr-style query construction, which only gets evaluated when needed. Can you provide some details on the use cases of your GB+ datasets? Commented Aug 22, 2021 at 9:29

2 Answers 2

2

I would suggest using the dbcooper found on github. https://github.com/chriscardillo/dbcooper

I have found huge improvements in speed when querying large datasets.

Firstly, Add your connection to your environment.

conn <- DBI::dbConnect(odbc::odbc(),
                   Driver = "",
                   Server = "",
                   Database = "",
                   UID="",
                   PWD="")

devtools::install_github("chriscardillo/dbcooper")
library(dbcooper)
dbcooper::dbc_init(con = conn, 
               con_id = "test", 
               tables = c("schema.table"))

This adds the function test_schema_table() to your environment which is used to call the data. To collect into your environment use scheme_table %>% collect()

Here is a microbenchmark I did to compare the results of both DBI and dbcooper.

mbm <- microbenchmark::microbenchmark(
  DBI = DBI::dbFetch(DBI::dbSendQuery(conn,qry)),
  dbcooper = ava_qry() %>% collect() ,  times=5
)

Here are the results of a microbenchmark I did to compare DBI with dbcooper.

Microbenchmark of DBI vs dbcooper

enter image description here

0

My answer includes use of a different package. I use RODBC which is found in cran at https://cran.r-project.org/web/packages/RODBC/index.html.

This has saved me SO MUCH frustration and wasted time that came from my previous method of exporting each query result to .csv to load it into my R environment. I found regular ODBC to be much slower than RODBC. I use the following functions:

sqlQuery() wraps the function that opens the connection to the SQL db with the first argument (in parentheses) and the query itself as the second argument. Put the query itself in quote marks.

odbcConnect() is itself the first argument in sqlquery(). The argument in odbcConnect() is the name of your connection to the SQL db. Put the connection name in quote marks.

odbcCloseAll() is the final function for this task set. Use this after each sqlQuery() to close the connection and save yourself from annoying warning messages.

Here is a simple example.

library(RODBC)
result <- sqlQuery(odbcConnect("ODBCConnectionName"),
           "SELECT * 
           FROM dbo.table 
           WHERE Collection_ID = 2498") 

odbcCloseAll()

Here is the same example PLUS data manipulation directly from the query result.

    library(dplyr)
    library(RODBC)

    result <- sqlQuery(odbcConnect("ODBCConnectionName"),
               "SELECT * 
               FROM dbo.table 
               WHERE Collection_ID = 2498") %>% 
        mutate(matchid = paste0(schoolID, "-", studentID)) %>% 
        distinct(matchid, .keep_all - TRUE)

     odbcCloseAll()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.