Questions tagged [zarr]

Zarr is a Python package providing an implementation of compressed, chunked, N-dimensional arrays, designed for use in parallel computing.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

ZARR: Inefficiency in Selecting Data Based on Bounding Box Coordinates with TIME Dimensions Only dataset

I have the following dataset which is a trajectory type dataset. LATITUDE and LONGITUDE are defined as coordinates in xarray BUT they aren't dimensions! All of the issues I've came across on ...
Laurent Besnard's user avatar
1 vote
0 answers
14 views

Do zarr arrays natively support integer scaling and offsets like NetCDF? If not, is there a workaround?

I have a bunch of NetCDF (.nc) files (ERA5 dataset) that I'm reading in Python through xarray and rioxarray. They end up as arrays of float32 (4 bytes) in memory. However, on disk they are stored as ...
Thomas's user avatar
  • 177k
0 votes
0 answers
42 views

indexing zarr with both coordinate arrays and slice

Currently zarr support advanced indexing by coordinate arrays: z = zarr.array(np.arange(3*4).reshape(3,4)) z.vindex[[0, 2], [1, 3]] array([ 1, 11]) But the coordinate arrays have to be provided for ...
Kang Liang's user avatar
1 vote
1 answer
105 views

Filter xarray ZARR dataset with GeoDataFrame

I am reading a ZARR file from a s3 bucket with xarray. I got to successfully filter by time and latitude/longitude: def read_zarr(self, dataset: str, region: Region) -> Any: # Read ZARR ...
Alvin Sartor's user avatar
  • 2,281
0 votes
1 answer
85 views

Upload zarr folder to Azure blob container

I tried to upload a zarr file (folder-like) to an Azure container using Python but it does not work properly, as it only uploaded the innermost files and deleted everything else in the container. This ...
Minh Phan's user avatar
0 votes
0 answers
65 views

Open zarr with xarray_tensorstore: deep copy is not working after openning file

I opened a file using xarray_tensorstore.open_zarr. While the opening was successful, I encountered an issue when trying to use copy(deep=True). However, copy(deep=True) works correctly when I use the ...
A-_-S's user avatar
  • 700
0 votes
0 answers
80 views

Compressing integers to 12-bit packed format with zarr and numcodecs

I am working with 12-bit integer image data (i.e. data aquired from a camera that uses a 12-bit ADC) which I ultimately store in a Zarr array. Currently I store the images as 16-bit integers, which ...
ptbrown's user avatar
  • 101
0 votes
1 answer
102 views

xarray Dataset resize coordinates in dimension for to_zarr()

I have a dataset and wish to append it to an existing zarr and I must handle that one dimension in the dataset, 'range', is of different size than in the zarr. I'm hoping to just pad data with zeros ...
vindo's user avatar
  • 1
0 votes
1 answer
57 views

How to store a subset of Xaray data into Zarr?

Context In the section Appending to existing Zarr stores, the example is as follows import xarray as xr import dask.array # Write zarr with empty structure dummies = dask.array.zeros(30, chunks=10) ...
Dahn's user avatar
  • 1,477
0 votes
0 answers
100 views

Initializing empty zarr for parallel writing

I want to select subset area from ERA5 Zarr and write it to my own bucket. I want to do parallel processing. I first read the data: import xarray, fsspec, boto3, dask import dask.array as da from dask....
Pörripeikko's user avatar
0 votes
0 answers
35 views

Change dtype of already existing zarr array

Could you tell me please if it is possible to change the dtype of already created zarr array? Best regards, Aliaksei
aaxx's user avatar
  • 55
0 votes
0 answers
103 views

aiohttp.client_exceptions.ClientConnectorError when using proxy (clash) when doownloading CMIP6 data through pangeo

1. Environment Linux (#44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux) Anaconda (23.1.0) running a proxy (clash) conda environment created by ...
Lorraine's user avatar
0 votes
1 answer
213 views

Xarray write large dataset on memory without killing the kernel

Context: I have the following dataset: Goal: I want to write it on my disk. I am using chunks so the dataset does not kill my kernel. Problem: I tried to save it on my disk with chunks using: Option ...
Nihilum's user avatar
  • 587
0 votes
0 answers
30 views

Why does zarr.convenince.consolidate_metadata() not work within a LinearFlow (metaflow)?

This is an issue about metaflow, zarr, python I am creating a LinearFlow using metaflow and zarr. All is going well except one key zarr function: when I try to consolidate all my metadata into a ...
Laure Vancau's user avatar
0 votes
1 answer
118 views

Interpolating values in xarray using non-indexed coordinates

I'm trying to fetch time series from geographical coordinates (single points) from Google ERA5 Reanalysis data. The dataset is following: import xarray data = xarray.open_zarr( 'gs://gcp-public-...
Pörripeikko's user avatar
0 votes
1 answer
261 views

How to reproject xarray dataset memory efficiently with chunks and dask?

Context: I have a netcdf file that I want to reproject. It is a costly operation, and I am learning how to use dask and zarr to do it efficiently without crashing my RAM. Code presentation: ds is a 3D ...
Nihilum's user avatar
  • 587
4 votes
1 answer
147 views

How to avoid reading half-written arrays spanning multiple chunks using zarr?

In a multiprocess situation, I want to avoid reading arrays from a zarr group that haven't fully finished writing by the other process yet. This functionality does not seem to come out of the box with ...
bluppfisk's user avatar
  • 2,640
2 votes
1 answer
322 views

can you save an object array using zarr?

Following zarr's tutorial, I'm trying to save a list of list of ints to a persistent zarr: Failed method 1: import numcodecs, zarr zarr.save("path/to/zarr", [[1], [2]], dtype=object, ...
David Taub's user avatar
1 vote
1 answer
86 views

Better way to identify chunks where data is available in zarr

I have a zarr store of weather data with 1 hr time interval for the year 2022. So 8760 chunks. But there are data only for random days. How do i check which are the hours in 0 to 8760, the data is ...
sjd's user avatar
  • 1,319
2 votes
1 answer
213 views

printing the uploading progress using `xarray.Dataset.to_zarr` function

I'm trying to upload an xarray dataset to GCP using the function ds.to_zarr(store=store), and it works perfect. However, I would like to show the progress of big datasets. Is there any option to chunk ...
Henry Ruiz's user avatar
0 votes
1 answer
240 views

How to prevent `to_zarr` method in xarray from writing all nan chunks to disk?

I want to save a very large zarr file (2 dimensional) chunked equally along both dimensions (X, X) occationally containing chunks made of all nans. To reduce the amount of chunks written to disk, I ...
thunderslash's user avatar
0 votes
0 answers
56 views

using a zarr store as a fixed-size buffer

I'm trying to use a zarr store as a fixed-size buffer (i.e. new data is appended to the end, and the same amount of data is removed from the beginning when a certain size is reached). The store is ...
bluppfisk's user avatar
  • 2,640
1 vote
0 answers
96 views

What is the maximum array size for Zarr?

I was not able to find information on any limits of Zarr arrays. Is there any practical limit on the size of Zarr arrays apart from the disk space? This is covering both: number of dimensions and size ...
SultanOrazbayev's user avatar
0 votes
1 answer
24 views

disable zarr.open() output in console

I have unwanted outputs in the console from zarr.open() method. It does not have 'verbose-like' parameter. How can I get rid of those input console ? I'm currently trying to open .ims files (Imaris ...
Willy Lutz's user avatar
0 votes
1 answer
458 views

How to create and return a Zarr file from xarray Dataset?

How would I go about creating and return a file new_zarr.zarr from a xarray Dataset? I know xarray.Dataset.to_zarr() exists but this returns a ZarrStore and I must return a bytes-like object. I have ...
A B's user avatar
  • 97
0 votes
1 answer
284 views

Slow reading of small zarr/S3 data through python-xarray inside a dockerized fastAPI app

I have a tiny dataset like this: <xarray.Dataset> Dimensions: (time: 24) Coordinates: * time (time) datetime64[ns] 2022-09-28 ... 2022-09-28T23:00:00 spatial_ref int64 0 Data ...
PierreL's user avatar
  • 169
0 votes
0 answers
379 views

Zarr slow read speed of 43.82 GB file with Xarray

I want to look-up 8760 times for a single lat/lon combo in less than a second from 43.82 GB file of wind data containing: 8760 times (every hour in a year) 721 latitudes (every 0.25° from -90.0° to ...
MischievousChild's user avatar
0 votes
1 answer
369 views

How to create Zarr array from a large stack of npy files?

I have a stack of 4 dimensional numpy arrays saved as .npy files. Each one is about 1.5 GB and I have 240 files, so about 360 GB total and much larger than memory. I want to combine them into a single ...
qsfzy's user avatar
  • 554
1 vote
1 answer
476 views

Initialize larger-than-memory Xarray Dataset

I would like to initialize a very large XArray dataset (as on-disk Zarr if possible) for later processing - various parts (spatial subsets) of the dataset will be populated by a different script. This ...
HyperCube's user avatar
  • 3,890
2 votes
1 answer
447 views

How to store data from dask.distributed on disk?

I'm trying to scale my computations from local Dask Arrays to Dask Distributed. Unfortunately, I am new to distributed computed, so I could not adapt the answer here for my purpose. Mainly my problem ...
Helmut's user avatar
  • 366
0 votes
1 answer
152 views

Add attributes to an existing Zarr storage

I have a zarr store that I open using xarray and zarr: report = xr.open_zarr(grid_file_name) where grid_file_name is pointing to a local zarr directory. I need to add some attribute to the store, and ...
Shofus's user avatar
  • 1
1 vote
1 answer
871 views

Using xarray to convert a zarr file to a netcdf causing memory allocation error

I have a zarr file that I'd like to convert to a netcdf which is too large to fit in memory. My computer has 32GB of RAM so writing ~5.5GB chunks shouldn't be a problem. However, within seconds of ...
Daniel Lassiter's user avatar
1 vote
1 answer
439 views

Open root zarr with multiple groups using xarray

Suppose I had a zarr file that has n groups each of which have only one zarr array and share at least 3 dimensions but may have others as well, how would I load an xarray Dataset from said zarr root ...
Curious's user avatar
  • 393
0 votes
1 answer
100 views

Disallow mpi4py to interfere with internal handling of mpi by a python API

I am using mpiexec on a cluster to run large-scale simulations using pyNEST (mpiexec -n $N python simulate.py). I export a large number of small files which often tends to exceed my inode quota on the ...
Ady's user avatar
  • 9
2 votes
2 answers
822 views

How to use Dask.Array.From_Zarr to open a zarr file on Dask?

I'm having quite a problem when converting a zarr file to a dask array. This is what I get when I type arr = da.from_zarr('gros.zarr/time') : but when I try on one coordinates such as time it works: ...
Severus's user avatar
  • 35
4 votes
1 answer
577 views

How to efficiently convert npy to xarray / zarr

I have a 37 GB .npy file that I would like to convert to Zarr store so that I can include coordinate labels. I have code that does this in theory, but I keep running out of memory. I want to use Dask ...
thomaskeefe's user avatar
  • 2,020
0 votes
1 answer
333 views

Speed up git add for binary files - disable compression

I am working on a new data structure that is version friendly. So I have a git repository inside a Zarr file. I don't have any push or upload data. just local version control. currently, the git add . ...
mzouink's user avatar
  • 497
0 votes
1 answer
563 views

Dask where returns NaN on valid array

I'm trying to accelerate my numpy code using dask. Following is a part of my numpy code arr_1 = np.load('<arr1_path>.npy') arr_2 = np.load('<arr2_path>.npy') arr_3 = np.load('<arr3_path&...
F Baig's user avatar
  • 349
0 votes
1 answer
838 views

How to convert Zarr data to GeoTiff?

I want to load the HRRR forecast data into Google Earth Engine, so I think I need to convert it to GeoTiff. e.g. import xarray as xr import s3fs fs = s3fs.S3FileSystem(anon=True) urls = ["s3://...
Adair's user avatar
  • 1,757
2 votes
1 answer
749 views

How to match all variables in xarray encoding (blosc, zarr compression)

The example of how to use zarr compression has the following code example see xarray doc: In [42]: import zarr In [43]: compressor = zarr.Blosc(cname="zstd", clevel=3, shuffle=2) In [44]: ...
marscher's user avatar
  • 820
0 votes
1 answer
519 views

zarr consolidate_metadata errors with TypeError: memoryview: a bytes-like object is required, not 'Array'

I'm trying to consolidate the metadata of an existing zarr store, though the same error occurs if I make a new zarr store and call zarr.consolidate_metadata(store). Code example: import zarr ## ...
Adair's user avatar
  • 1,757
0 votes
1 answer
1k views

xarray loading int data as float

Say I create a dataset with an integer variable. import xarray as xr import numpy as np int_var = np.random.randint(0, 10, 10) ds = xr.Dataset(data_vars={"int_var": (("x"), ...
Adair's user avatar
  • 1,757
5 votes
0 answers
847 views

Storing Dask Array using Zarr Consumes Too Much Memory

I have a long list of .zarr arrays, that I would like to merge into a single array and write to disk. My code approximately looks as follows: import dask.array import zarr import os local_paths = ['...
r0f1's user avatar
  • 2,757
0 votes
1 answer
504 views

Is it possible to store multiple groups and arrays in a single file?

I've been using HDF5 to store time series data and I want to try using Zarr due to its various features. I'm reading its tutorial and following each step, and I've realized that maybe Zarr uses ...
maynull's user avatar
  • 1,946
5 votes
1 answer
8k views

ValueError: unrecognized engine zarr must be one of: ['scipy', 'store']

I am trying to open zarr file as, import pandas as pd import xarray as xr xf = xr.open_zarr("../../data/processed/geolink_norge_dataset/geolink_norge_well_logs.zarr") But there comes out ...
K Code's user avatar
  • 51
0 votes
2 answers
267 views

How can I rename a Zarr array without writing new store?

I have a Zarr datastore, but I need to rename one of the dimensions. Let's say I have this (from xarray docs): data = np.random.rand(4, 3) locs = ["IA", "IL", "IN"] times ...
j sad's user avatar
  • 1,065
0 votes
2 answers
534 views

Transform zarr directory storage to zip storage

codes: store = zarr.ZipStore("/mnt/test.zip", "r") Problem description: Hi, sry for bothering, I found this statement inside Zarr official documentation about ZipStorage: ...
eddie Gao's user avatar
1 vote
1 answer
1k views

zarr not respecting chunk size from xarray and reverting to original chunk size

I'm opening a zarr file and then rechunking it and then writing it back out to a different zarr store. Yet when I open it back up it doesn't respect the chunk size I previously wrote. Here is the code ...
clifgray's user avatar
  • 4,349
2 votes
2 answers
690 views

Does Zarr has built-in multi-threading support for fast read and write?

I am trying to speed up reading and writing Zarr files using multi-threading. For example, if I can store an array in 5 chunks, is there a way to use a thread per chunk to speed up reading and writing ...
Ali Jooya's user avatar
0 votes
1 answer
1k views

xarray.Dataset.to_zarr: docs for “Appending to existing Zarr stores”?

In the description of the 'region' argument to xarray.Dataset.to_zarr the last sentence states: See “Appending to existing Zarr stores” in the reference documentation for full details. I have not ...
user15699989's user avatar