I'm trying to extract a subset of depth data from GEBCO's global ocean bathymetry dataset, which is a 10.9gb .nc
file, netcdf4
(direct link).
I open a connection to the file, which doesn't load it into memory:
library(ncdf4)
GEBCO <- nc_open(filename = "GEBCO_2019.nc", verbose = T)
Find the lat & lon indices corresponding to my subset area:
LonIdx <- which(GEBCO$dim$lon$vals < -80 & GEBCO$dim$lon$vals > -81.7) #n=408 long
LatIdx <- which(GEBCO$dim$lat$vals < 26 & GEBCO$dim$lat$vals > 25) #n=240; 240*408=97920
Then get Z data for those extents:
z <- ncvar_get(GEBCO, GEBCO$var$elevation)[LonIdx, LatIdx]
Resulting in:
Error: cannot allocate vector of size 27.8gb
However it does this regardless of the size of the subset, even down to a 14*14 matrix. I presume therefore that ncvar_get()
is pulling the whole database in order to extract the indices... even though I was under the impression that the entire point of netcdf
files was that you could extract using matrix indexing without loading the whole thing to memory?
FWIW I'm on a 32gb linux machine, so it should work anyway? [edit, and the file is 10.9gb in the first place, so one would think a subset would be smaller]
Any ideas/intel/insights gratefully received. Thanks in advance.
Edit: other times it crashes RStudio rather than giving the error. R Session Aborted, fatal error, session terminated. RAM usage was: