2

I've run into a very strange situation when I load a csv file into R Studio. When I try loading a file with ~190k records, only ~8k records show up in my dataframe.

I tried loading the data into R Studio on my Mac and everything works fine. I Googled the issue but wasn't able to find a solution.

Below are the specs of both machines:

PC - R + R Studio: latest version -

RAM: 8GB

MAC - R + R Studio: latest version -

RAM: 16GB

I've also tried various solutions like (quote = "") but this did not resolve the truncation. Perhaps I'm missing something very simple. Any ideas?

3
  • 1
    Could you provide some code showing this issue, how you've loaded in the data, how you've checked the size. Perhaps an nrow() so we can see whats going on or a screen shot. Seems strange though.
    – Badger
    Sep 17, 2015 at 22:32
  • 3
    What does your loading statement look like? Maybe give us a sample row from your dataset? perhaps do str on the loaded data frame and paste the output? If there is an error message after loading, could you provide it?
    – Michal
    Sep 17, 2015 at 22:32
  • Thanks all for responding. I tried loading data both manually and through a standard 'read.csv' statement on both machines with my PC failing to load the complete data set. I figured if both methods (manual + code) produced the same results the issue maybe more R Studio configuration or perhaps my hardware configuration. I'll try to replicate the issue and provide more context. Thanks again! Oct 28, 2015 at 21:49

1 Answer 1

4

Try to use fread function which can read large data more quickly than read.csv. In my machine with R + Rstudio, I can read about 3G csv data under 4G RAM. For example:

library("data.table")
data <- fread("ab.csv", header=TRUE, sep=",", colClasses = "numeric")

BTW, figure out the memory limit and extend max number of your R by following commands:

memory.limit()
# set max memory usage is 2G
memory.size(max=2000)
2
  • After trying this solution my machine was able to read in everything correctly. Thanks again! Nov 16, 2015 at 22:42
  • to add, I still do not know the root cause of why the read.csv command is truncating the number of rows loaded into my global environment on my Windows machine and not my Mac. Nov 16, 2015 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.