2

I'm trying to extract a subset of depth data from GEBCO's global ocean bathymetry dataset, which is a 10.9gb .nc file, netcdf4 (direct link).

I open a connection to the file, which doesn't load it into memory:

library(ncdf4)
GEBCO <- nc_open(filename = "GEBCO_2019.nc", verbose = T)

Find the lat & lon indices corresponding to my subset area:

LonIdx <- which(GEBCO$dim$lon$vals < -80 & GEBCO$dim$lon$vals > -81.7) #n=408 long
LatIdx <- which(GEBCO$dim$lat$vals < 26 & GEBCO$dim$lat$vals > 25) #n=240; 240*408=97920

Then get Z data for those extents:

z <- ncvar_get(GEBCO, GEBCO$var$elevation)[LonIdx, LatIdx]

Resulting in:

Error: cannot allocate vector of size 27.8gb

However it does this regardless of the size of the subset, even down to a 14*14 matrix. I presume therefore that ncvar_get() is pulling the whole database in order to extract the indices... even though I was under the impression that the entire point of netcdf files was that you could extract using matrix indexing without loading the whole thing to memory?

FWIW I'm on a 32gb linux machine, so it should work anyway? [edit, and the file is 10.9gb in the first place, so one would think a subset would be smaller]

Any ideas/intel/insights gratefully received. Thanks in advance.

Edit: other times it crashes RStudio rather than giving the error. R Session Aborted, fatal error, session terminated. RAM usage was:

netcdfRAMfail

1 Answer 1

4

Ok, solved. Turns out the answer I found online before using [LonIdx, LatIdx] indexes the object after the whole thing is read to memory. Notwithstanding I still don't know why this was a problem given its filesize is a third my memory, and failing expanded size is within my memory, this is still the wrong way to go.

Assuming ones rows and columns are contiguous (they should be in netcdf) the solution is:

z <- ncvar_get(nc = GEBCO,
               varid = GEBCO$var$elevation,
               start = c(LonIdx[1],
                         LatIdx[1]),
               count = c(length(LonIdx),
                         length(LatIdx)),
               verbose = T)

To convert to long format:

lon <- GEBCO$dim$lon$vals[LonIdx]
lat <- GEBCO$dim$lat$vals[LatIdx]
rownames(z) <- as.character(lon)
colnames(z) <- as.character(lat)
library(tidyr)
library(magrittr)
ztbl <- as_tibble(z, rownames = "lon")
ztbl %<>% pivot_longer(-lon, names_to = "lat", values_to = "depth")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.