Questions tagged [zarr]
Zarr is a Python package providing an implementation of compressed, chunked, N-dimensional arrays, designed for use in parallel computing.
96
questions
0
votes
0
answers
14
views
ZARR: Inefficiency in Selecting Data Based on Bounding Box Coordinates with TIME Dimensions Only dataset
I have the following dataset which is a trajectory type dataset.
LATITUDE and LONGITUDE are defined as coordinates in xarray BUT they aren't dimensions! All of the issues I've came across on ...
1
vote
0
answers
14
views
Do zarr arrays natively support integer scaling and offsets like NetCDF? If not, is there a workaround?
I have a bunch of NetCDF (.nc) files (ERA5 dataset) that I'm reading in Python through xarray and rioxarray. They end up as arrays of float32 (4 bytes) in memory.
However, on disk they are stored as ...
0
votes
0
answers
42
views
indexing zarr with both coordinate arrays and slice
Currently zarr support advanced indexing by coordinate arrays:
z = zarr.array(np.arange(3*4).reshape(3,4))
z.vindex[[0, 2], [1, 3]]
array([ 1, 11])
But the coordinate arrays have to be provided for ...
1
vote
1
answer
105
views
Filter xarray ZARR dataset with GeoDataFrame
I am reading a ZARR file from a s3 bucket with xarray. I got to successfully filter by time and latitude/longitude:
def read_zarr(self, dataset: str, region: Region) -> Any:
# Read ZARR ...
0
votes
1
answer
85
views
Upload zarr folder to Azure blob container
I tried to upload a zarr file (folder-like) to an Azure container using Python but it does not work properly, as it only uploaded the innermost files and deleted everything else in the container. This ...
0
votes
0
answers
65
views
Open zarr with xarray_tensorstore: deep copy is not working after openning file
I opened a file using xarray_tensorstore.open_zarr. While the opening was successful, I encountered an issue when trying to use copy(deep=True). However, copy(deep=True) works correctly when I use the ...
0
votes
0
answers
80
views
Compressing integers to 12-bit packed format with zarr and numcodecs
I am working with 12-bit integer image data (i.e. data aquired from a camera that uses a 12-bit ADC) which I ultimately store in a Zarr array. Currently I store the images as 16-bit integers, which ...
0
votes
1
answer
102
views
xarray Dataset resize coordinates in dimension for to_zarr()
I have a dataset and wish to append it to an existing zarr and I must handle that one dimension in the dataset, 'range', is of different size than in the zarr. I'm hoping to just pad data with zeros ...
0
votes
1
answer
57
views
How to store a subset of Xaray data into Zarr?
Context
In the section Appending to existing Zarr stores, the example is as follows
import xarray as xr
import dask.array
# Write zarr with empty structure
dummies = dask.array.zeros(30, chunks=10)
...
0
votes
0
answers
100
views
Initializing empty zarr for parallel writing
I want to select subset area from ERA5 Zarr and write it to my own bucket. I want to do parallel processing.
I first read the data:
import xarray, fsspec, boto3, dask
import dask.array as da
from dask....
0
votes
0
answers
35
views
Change dtype of already existing zarr array
Could you tell me please if it is possible to change the dtype of already created zarr array?
Best regards,
Aliaksei
0
votes
0
answers
103
views
aiohttp.client_exceptions.ClientConnectorError when using proxy (clash) when doownloading CMIP6 data through pangeo
1. Environment
Linux (#44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux)
Anaconda (23.1.0)
running a proxy (clash)
conda environment created by ...
0
votes
1
answer
213
views
Xarray write large dataset on memory without killing the kernel
Context:
I have the following dataset:
Goal: I want to write it on my disk. I am using chunks so the dataset does not kill my kernel.
Problem:
I tried to save it on my disk with chunks using:
Option ...
0
votes
0
answers
30
views
Why does zarr.convenince.consolidate_metadata() not work within a LinearFlow (metaflow)?
This is an issue about metaflow, zarr, python
I am creating a LinearFlow using metaflow and zarr.
All is going well except one key zarr function: when I try to consolidate all my metadata into a ...
0
votes
1
answer
118
views
Interpolating values in xarray using non-indexed coordinates
I'm trying to fetch time series from geographical coordinates (single points) from Google ERA5 Reanalysis data. The dataset is following:
import xarray
data = xarray.open_zarr(
'gs://gcp-public-...
0
votes
1
answer
261
views
How to reproject xarray dataset memory efficiently with chunks and dask?
Context: I have a netcdf file that I want to reproject. It is a costly operation, and I am learning how to use dask and zarr to do it efficiently without crashing my RAM.
Code presentation: ds is a 3D ...
4
votes
1
answer
147
views
How to avoid reading half-written arrays spanning multiple chunks using zarr?
In a multiprocess situation, I want to avoid reading arrays from a zarr group that haven't fully finished writing by the other process yet. This functionality does not seem to come out of the box with ...
2
votes
1
answer
322
views
can you save an object array using zarr?
Following zarr's tutorial, I'm trying to save a list of list of ints to a persistent zarr:
Failed method 1:
import numcodecs, zarr
zarr.save("path/to/zarr", [[1], [2]], dtype=object, ...
1
vote
1
answer
86
views
Better way to identify chunks where data is available in zarr
I have a zarr store of weather data with 1 hr time interval for the year 2022. So 8760 chunks. But there are data only for random days. How do i check which are the hours in 0 to 8760, the data is ...
2
votes
1
answer
213
views
printing the uploading progress using `xarray.Dataset.to_zarr` function
I'm trying to upload an xarray dataset to GCP using the function ds.to_zarr(store=store), and it works perfect. However, I would like to show the progress of big datasets. Is there any option to chunk ...
0
votes
1
answer
240
views
How to prevent `to_zarr` method in xarray from writing all nan chunks to disk?
I want to save a very large zarr file (2 dimensional) chunked equally along both dimensions (X, X) occationally containing chunks made of all nans. To reduce the amount of chunks written to disk, I ...
0
votes
0
answers
56
views
using a zarr store as a fixed-size buffer
I'm trying to use a zarr store as a fixed-size buffer (i.e. new data is appended to the end, and the same amount of data is removed from the beginning when a certain size is reached).
The store is ...
1
vote
0
answers
96
views
What is the maximum array size for Zarr?
I was not able to find information on any limits of Zarr arrays. Is there any practical limit on the size of Zarr arrays apart from the disk space? This is covering both: number of dimensions and size ...
0
votes
1
answer
24
views
disable zarr.open() output in console
I have unwanted outputs in the console from zarr.open() method. It does not have 'verbose-like' parameter. How can I get rid of those input console ?
I'm currently trying to open .ims files (Imaris ...
0
votes
1
answer
458
views
How to create and return a Zarr file from xarray Dataset?
How would I go about creating and return a file new_zarr.zarr from a xarray Dataset?
I know xarray.Dataset.to_zarr() exists but this returns a ZarrStore and I must return a bytes-like object.
I have ...
0
votes
1
answer
284
views
Slow reading of small zarr/S3 data through python-xarray inside a dockerized fastAPI app
I have a tiny dataset like this:
<xarray.Dataset>
Dimensions: (time: 24)
Coordinates:
* time (time) datetime64[ns] 2022-09-28 ... 2022-09-28T23:00:00
spatial_ref int64 0
Data ...
0
votes
0
answers
379
views
Zarr slow read speed of 43.82 GB file with Xarray
I want to look-up 8760 times for a single lat/lon combo in less than a second from 43.82 GB file of wind data containing:
8760 times (every hour in a year)
721 latitudes (every 0.25° from -90.0° to ...
0
votes
1
answer
369
views
How to create Zarr array from a large stack of npy files?
I have a stack of 4 dimensional numpy arrays saved as .npy files. Each one is about 1.5 GB and I have 240 files, so about 360 GB total and much larger than memory. I want to combine them into a single ...
1
vote
1
answer
476
views
Initialize larger-than-memory Xarray Dataset
I would like to initialize a very large XArray dataset (as on-disk Zarr if possible) for later processing - various parts (spatial subsets) of the dataset will be populated by a different script.
This ...
2
votes
1
answer
447
views
How to store data from dask.distributed on disk?
I'm trying to scale my computations from local Dask Arrays to Dask Distributed.
Unfortunately, I am new to distributed computed, so I could not adapt the answer here for my purpose.
Mainly my problem ...
0
votes
1
answer
152
views
Add attributes to an existing Zarr storage
I have a zarr store that I open using xarray and zarr:
report = xr.open_zarr(grid_file_name)
where grid_file_name is pointing to a local zarr directory.
I need to add some attribute to the store, and ...
1
vote
1
answer
871
views
Using xarray to convert a zarr file to a netcdf causing memory allocation error
I have a zarr file that I'd like to convert to a netcdf which is too large to fit in memory. My computer has 32GB of RAM so writing ~5.5GB chunks shouldn't be a problem. However, within seconds of ...
1
vote
1
answer
439
views
Open root zarr with multiple groups using xarray
Suppose I had a zarr file that has n groups each of which have only one zarr array and share at least 3 dimensions but may have others as well, how would I load an xarray Dataset from said zarr root ...
0
votes
1
answer
100
views
Disallow mpi4py to interfere with internal handling of mpi by a python API
I am using mpiexec on a cluster to run large-scale simulations using pyNEST (mpiexec -n $N python simulate.py). I export a large number of small files which often tends to exceed my inode quota on the ...
2
votes
2
answers
822
views
How to use Dask.Array.From_Zarr to open a zarr file on Dask?
I'm having quite a problem when converting a zarr file to a dask array. This is what I get when I type arr = da.from_zarr('gros.zarr/time') :
but when I try on one coordinates such as time it works:
...
4
votes
1
answer
577
views
How to efficiently convert npy to xarray / zarr
I have a 37 GB .npy file that I would like to convert to Zarr store so that I can include coordinate labels. I have code that does this in theory, but I keep running out of memory. I want to use Dask ...
0
votes
1
answer
333
views
Speed up git add for binary files - disable compression
I am working on a new data structure that is version friendly.
So I have a git repository inside a Zarr file.
I don't have any push or upload data. just local version control.
currently, the git add . ...
0
votes
1
answer
563
views
Dask where returns NaN on valid array
I'm trying to accelerate my numpy code using dask. Following is a part of my numpy code
arr_1 = np.load('<arr1_path>.npy')
arr_2 = np.load('<arr2_path>.npy')
arr_3 = np.load('<arr3_path&...
0
votes
1
answer
838
views
How to convert Zarr data to GeoTiff?
I want to load the HRRR forecast data into Google Earth Engine, so I think I need to convert it to GeoTiff.
e.g.
import xarray as xr
import s3fs
fs = s3fs.S3FileSystem(anon=True)
urls = ["s3://...
2
votes
1
answer
749
views
How to match all variables in xarray encoding (blosc, zarr compression)
The example of how to use zarr compression has the following code example see xarray doc:
In [42]: import zarr
In [43]: compressor = zarr.Blosc(cname="zstd", clevel=3, shuffle=2)
In [44]: ...
0
votes
1
answer
519
views
zarr consolidate_metadata errors with TypeError: memoryview: a bytes-like object is required, not 'Array'
I'm trying to consolidate the metadata of an existing zarr store, though the same error occurs if I make a new zarr store and call zarr.consolidate_metadata(store).
Code example:
import zarr
## ...
0
votes
1
answer
1k
views
xarray loading int data as float
Say I create a dataset with an integer variable.
import xarray as xr
import numpy as np
int_var = np.random.randint(0, 10, 10)
ds = xr.Dataset(data_vars={"int_var": (("x"), ...
5
votes
0
answers
847
views
Storing Dask Array using Zarr Consumes Too Much Memory
I have a long list of .zarr arrays, that I would like to merge into a single array and write to disk.
My code approximately looks as follows:
import dask.array
import zarr
import os
local_paths = ['...
0
votes
1
answer
504
views
Is it possible to store multiple groups and arrays in a single file?
I've been using HDF5 to store time series data and I want to try using Zarr due to its various features.
I'm reading its tutorial and following each step, and I've realized that maybe Zarr uses ...
5
votes
1
answer
8k
views
ValueError: unrecognized engine zarr must be one of: ['scipy', 'store']
I am trying to open zarr file as,
import pandas as pd
import xarray as xr
xf = xr.open_zarr("../../data/processed/geolink_norge_dataset/geolink_norge_well_logs.zarr")
But there comes out ...
0
votes
2
answers
267
views
How can I rename a Zarr array without writing new store?
I have a Zarr datastore, but I need to rename one of the dimensions. Let's say I have this (from xarray docs):
data = np.random.rand(4, 3)
locs = ["IA", "IL", "IN"]
times ...
0
votes
2
answers
534
views
Transform zarr directory storage to zip storage
codes:
store = zarr.ZipStore("/mnt/test.zip", "r")
Problem description:
Hi, sry for bothering, I found this statement inside Zarr official documentation about ZipStorage:
...
1
vote
1
answer
1k
views
zarr not respecting chunk size from xarray and reverting to original chunk size
I'm opening a zarr file and then rechunking it and then writing it back out to a different zarr store. Yet when I open it back up it doesn't respect the chunk size I previously wrote. Here is the code ...
2
votes
2
answers
690
views
Does Zarr has built-in multi-threading support for fast read and write?
I am trying to speed up reading and writing Zarr files using multi-threading. For example, if I can store an array in 5 chunks, is there a way to use a thread per chunk to speed up reading and writing ...
0
votes
1
answer
1k
views
xarray.Dataset.to_zarr: docs for “Appending to existing Zarr stores”?
In the description of the 'region' argument to xarray.Dataset.to_zarr the last sentence states: See “Appending to existing Zarr stores” in the reference documentation for full details. I have not ...