Questions tagged [dask]

Dask is a parallel computing and data analytics library for Python. It supports dynamic task scheduling optimized for computation as well as big data collections.

Filter by
Sorted by
Tagged with
0 votes
1 answer
25 views

Dask dataframe condition to compare column doesn't working

I'm trying to compare the column names of a dask dataframe to then change the column data type, however my condition never is true: column_name = "names" print(f"Col Name: {column_name} ...
FreddicMatters's user avatar
0 votes
0 answers
17 views

How to further partition/split an already partitioned dask dataframe

I have a very large DataFrame, that I partition based on values in one column "A" using the dask.DataFrame.set_index() method. Such N partitions are still too large to fit into memory when ...
Ste's user avatar
  • 53
0 votes
0 answers
10 views

User-defined types for groupby apply with p2p shuffling

It seems that p2p shuffling is not supported for a dask dataframe that contains columns of user-defined types. This is not a limitation for tasks shuffling though. Here’s an example: import dask ...
Yilie Ma's user avatar
0 votes
0 answers
22 views

Connection problems with dask scheduler

I've set up a kubernetes cluster with GKE and installed the dask-kubernetes-operator. When i try to start the cluster like this cluster: KubeCluster = KubeCluster(custom_cluster_spec="cluster....
TuemmlerTanne11's user avatar
0 votes
0 answers
23 views

Dask dataframes: read_excel file by file with panda

I'm processing multiple different types of files in different sizes with different columns. Files aren't too big (max 5MB and 20-30k rows) but panda was struggling with so decided to rewrite it with ...
niebyl2002's user avatar
0 votes
0 answers
19 views

I am trying to use geopandas to find polygons with are within 0.5m of each other using Dask. How do I multithread this operation?

so the current code I have can find intersecting cells, but Dask will not run it in parallel - is there a reason why, and is there a simpler and faster way to do this? The error message is located at ...
Reuben's user avatar
  • 81
0 votes
0 answers
29 views

Dask dashboard doesn't show any processing activity until after Xarray datasets are written out

I have a program that reads in several Xarray datasets, does some merging, reordering, concatenating to produce several merged Xarray datasets, and then writes them out to several files. I read in the ...
Clay's user avatar
  • 45
0 votes
0 answers
13 views

dask sshcluster results in: RuntimeError: Cluster failed to start: Worker failed to start

I try to get dask sshcluster (https://docs.dask.org/en/latest/deploying-ssh.html) up and running results in: RuntimeError: Cluster failed to start: Worker failed to start Python = = 3.11.6 Dask == ...
Jürgen P.'s user avatar
0 votes
0 answers
35 views

Error while trying to compute Dask dataframe from sql

I created a Dask dataframe using read_sql_query method and when I try to compute (some operation like pivoting, require computing), it raises an SQL error stating that it did not find the column I ...
dsnishimura's user avatar
0 votes
1 answer
33 views

Why aren't modifications made to a python module's variables propagating to new parallel processes?

I have an embarrassingly parallel list of tasks I want to perform. Currently I'm importing the configuration for these tasks as a module. example one-line configuration.py: result_folder = "...
OneTimeShot's user avatar
0 votes
1 answer
33 views

Dask array operation on chunks with no return

I have a dask array of dimension (C, Y, X) (for instance, (100, 50000, 50000)). I want to perform an operation on each chunk that will add some value to a small numpy array, that we will call x. Note ...
Quentin BLAMPEY's user avatar
0 votes
1 answer
54 views

Resampling huge Pandas Dataframe throws ArrayMemoryError

At work, we have a machine, that is logging some sensor values (e.g. temperatures, pressures, speed, etc.). I have a csv file with these values with somewhat in between 6 and 7 million lines and 13 ...
user22906295's user avatar
0 votes
1 answer
18 views

dask read_csv is fast but dataframe operation is slow

I am trying to improve speed of read_csv() then later dataframe using pandas 2. I tried dask today and read_csv() is indeed really fast. But dataframe operation is slow. why is that? how to improve ...
roudan's user avatar
  • 3,398
1 vote
1 answer
17 views

AttributeError: module 'dask' has no attribute 'set_options' after installing dask with conda

I am using annaconda and I just installed dask using below https://anaconda.org/conda-forge/dask conda install -c conda-forge dask I can use read_csv() and no error, however, I got this error . I am ...
roudan's user avatar
  • 3,398
1 vote
1 answer
33 views

dask import errors, dataframe/client - version conflicts with pandas?

Not all versions of dask.dataframe and pandas are compatible. This has been already addressed in this question I have tried several combinations but in combinations with more recent dask versions, ...
Waterkant's user avatar
  • 265
0 votes
1 answer
21 views

Dask LocalCluster creating threads inside subprocesses

I'm new to dask and I seem to have some issues with something that should be fairly straight forward so I'm sure I'm missing something. I have a program that supposed to poll messages from kafka, then ...
Ema Il's user avatar
  • 325
0 votes
0 answers
16 views

Stitching together irregular rasters into an XArray dataset that's too large for memory

I'm working on a project that involves many layers of raster data over a large area. This data is currently being stored in a set of raster files, each covering a small irregular polygon. I'd like ...
ghostm's user avatar
  • 1
0 votes
0 answers
31 views

rasterio doesn't import to local dask cluster

I am running an analysis on a large number of rasters with Dask in a Jupyter Lab notebook on a Windows computer. When I run my analysis using Coiled (cloud cluster), the analysis runs fine. However, ...
DGibbs's user avatar
  • 13
0 votes
2 answers
45 views

dask: dtype mismatch in read_json when convert-string=False

When reading a JSON-lines file that contains complex fields with dask I am unable to get the correct data types. The JSON-lines file looks like this {"col1": "some text", "...
raywib's user avatar
  • 50
0 votes
0 answers
39 views

Dask DataFrame: write multiple CSV by column

I have a Dask DataFrame with a column "file". I want to write each line of the dataframe to a CSV whose path is given by the "file" column. For instance, on the example below, the ...
Quentin BLAMPEY's user avatar
0 votes
1 answer
32 views

Issue while using map_partitions in dask with a power function on pandas table with list elements

I am having the following code in python developed using the Dask framework: # Create a Pandas DataFrame df = pd.DataFrame({ 'A': [[1], [2], [3], [4], [5]], 'B': [[6], [7], [8], [9], [10]] }) ...
naseefo's user avatar
  • 722
-2 votes
0 answers
23 views

Unexpected argument flush in dask

I am running a code to analyze my data and I keep getting this error that I cant resolve yet after trying for several days. I ran this code previously with no issues. here is my code: %%time ...
Halake's user avatar
  • 9
0 votes
0 answers
47 views

Large memory usage when calculating std dev over rolling window

I have a 2D array, and I want to calculate the mean and standard deviation over a spatial window of win_size pixels (~50 pixels or more). I only need to extract this for a subset of pixels whose ...
Jose's user avatar
  • 2,099
0 votes
0 answers
16 views

Migration Prefect1-2 and multithreading

Please, need your help. I am seeking a way to migrate my quiet complexity Flow code from Prefect-1 to Prefect-2. Flow worked perfectly on the old Prefect version. But when I remade it to 2-nd version ...
Pavel_K's user avatar
  • 13
0 votes
1 answer
32 views

How to persist a dask dataframe loaded with dask.dataframe.from_delayed

I have a large sharded dataset stored in a custom format that would benefit greatly from dask.dataframe.from_delayed However, I'm seeing odd behavior when trying to persist the resulting dataframe: ...
kjleftin's user avatar
  • 713
0 votes
0 answers
47 views

Typing when passing xarray DataArray objects to numpy ufuncs

I have a function with type annotations that looks like the following: import xarray as xr import numpy as np def compute_relative_azimuth(sat_azi: xr.DataArray, sun_azi: xr.DataArray) -> xr....
djhoese's user avatar
  • 3,612
0 votes
1 answer
16 views

Dask efficient indexing of column-stored data as dask arrays

I'm trying to find an efficient way to index a dataset that is stored by columns in zarr arrays loaded with Dask (Python). I have to perform some operations on it, among which indexing all the columns ...
stavoltafunzia's user avatar
0 votes
0 answers
30 views

Dask client causing memory expection?

I´ve been migrating some of my codes from pandas do dask. Many of them are already working. Studing a little bit more about dask, I´ve faced it "Client()" feature for monitoring usage. Nice ...
FábioRB's user avatar
  • 335
0 votes
1 answer
52 views

How to use large table from exasol database for statistical analysis in Python?

I have a table with 36 millions rows and i need to run different statistical analysis (e.g. hypohtesis tests, distribution analysis etc.). Since i get a memory error when i use the export_to_pandas ...
dika's user avatar
  • 63
0 votes
1 answer
61 views

Dask dataframe -containing columns that are numpy arrays- conversion to parquet error

I have a dask dataframe which I am trying to convert to parquet files. This dataframe has columns which are numpy arrays, but column type is "object". When I try to do this: name_function = ...
Ahmed Gamal's user avatar
0 votes
1 answer
29 views

close dask client and cluster when catching any errors or exceptions

I am writing a python function to do data processing with dask. I want to automatically close the dask cluster and client if any errors or exceptions raised. So I use the with ... as: statement. The ...
Kang Liang's user avatar
0 votes
0 answers
21 views

How to tackle Dask unmanaged memory in Windows OS when using delayed functions?

I have the below traditional Python function, without any array-type flavour, but which I need to run many times. Hence, I used Dask-parallelization using dask.delayed. However, I can see a gradual ...
Anirban Chakraborty's user avatar
0 votes
0 answers
36 views

How to turn a dask scalar of type boolean into a boolean expression

I have a long code with many data manipulations in which at the end of it I want to get a boolean expression from comparing two dask series, this is the final part of my code: ...
Mika Bell's user avatar
0 votes
1 answer
27 views

Dask from_delayed() causing high memory usage

I'm following the instructions from (https://docs.dask.org/en/stable/delayed-collections.html) to create a custom data loader for a Dask DataFrame, which is basically this: import dask.dataframe as dd ...
Tarantula's user avatar
  • 19.2k
0 votes
1 answer
34 views

Simple transformation in Dask data frame produces unexpected result

I have what looks to be a simple task: Compute the grouped cumulative ('running') minimum of Dask DataFrame column. My data frame looks like this: import pandas as pd import numpy as np import dask....
Skumin's user avatar
  • 175
0 votes
0 answers
26 views

'TypeError: Categorical.ordered must be the same' when computing dask dataframe after pivot_table

I have a Dask DataFrame with the following structure: Dask DataFrame Structure: A B C npartitions=25 ...
Iván Martínez's user avatar
0 votes
0 answers
26 views

xarray Can't read closed raster file ValueError by using Planetary Computer

I would like to use the OTCI dataset from the Planetary Computer Sentinel-3 Land dataset, by accessning it with the "stacstack" api, convert it to an xarray DataArray and do some further ...
Dávid's user avatar
  • 1
0 votes
0 answers
29 views

Dask quantile over grouped data

Given a Dask dataframe with columns "eco_regions" and "ndvi", is there any way to group the data by the eco_regions and then compute the 90th percentile (0.9 quantile) of the ndvi ...
Cursore's user avatar
  • 33
2 votes
1 answer
42 views

How to use Dask to parallelize up a `sel()` operation in xarray?

I have an array of values called speed, and I'm mapping it to another array of values of the same shape called power by looking up the nearest value in a lookup table speed_to_power_lut. This process ...
Thomas's user avatar
  • 176k
-1 votes
1 answer
41 views

How to specify correct dtype for column of lists when creating a dask dataframe from_pandas?

When creating a dask Dataframe with the from_pandas method, the formerly correct dtype object becomes a string[pyarrow]. import dask.dataframe as dd import pandas as pd df = pd.DataFrame( { ...
spettekaka's user avatar
0 votes
1 answer
60 views

I used Dask to read my 7GB CSV but now there is an error

The error says "NotImplementedError: dd.DataFrame.apply only supports axis=1 Try: df.apply(func, axis=1)" Here is my line of code: # Read the CSV file in using dask import dask.dataframe as ...
Krizel Delos Reyes's user avatar
0 votes
1 answer
21 views

Parallelize function over 3D array chunked with Dask

I have a dataset ds (dimension mid_date,y,x) that hosts a 3D numpy array (ds.v). I have a function that I want to apply on each y,x cell of the array, which returns a vector reduced in dimension ...
Nihilum's user avatar
  • 577
0 votes
0 answers
117 views

Error about field having incompatible type when reading parquet with dask.dataframe.read_parquet

I am using jupyter/scipy-notebook, with dask version: >>> dask.__version__ '2023.9.3' When reading this parquet file, using dask.dataframe.read_parquet, I get an error. mme = dask.dataframe....
dkjackyu's user avatar
0 votes
0 answers
13 views

Dask task stream finishing much faster than actual computation

I am attempting to run a process in parallel using dask.bag, but the process is taking longer than the task stream seems to suggest. I am on dask version 2023.9.3 I am on a single machine ...
skrhee's user avatar
  • 348
0 votes
2 answers
64 views

Regex Field Match and Replace in a network-Python

I have a large csv (+1000000 rows) that I need to do a regex search and replace function on. In short, I need to take two columns and find matches between them; and then use the matched rows to ...
Jackson Dunn's user avatar
0 votes
0 answers
28 views

dask.read_csv much slower than pandas.read_csv with 50k columns

I have a csv with 1k rows and 50k columns. I takes 9s to calculate the min of all columns with pandas panda_df = pd.read_csv(path, sep=",") # 8s mins = panda_df.min(numeric_only=...
Daniel Pinyol's user avatar
0 votes
0 answers
45 views

Convert series of lists into dataframe with multiple columns

How do I convert a Dask series that contains lists of varying length into a dataframe with multiple columns? In Pandas it's: import pandas as pd ps = pd.Series([["a0", "a1"], [&...
spettekaka's user avatar
0 votes
0 answers
42 views

"Failed to deserialize" when using dask.dataframe.from_delayed on a collection of pandas DataFrames

Describe the issue: I'm currently trying to read from a database in parallel using Dask and Pandas. I'm creating a collection of Dask delayed objects using pd.read_sql and then using Dask's ...
Arjun Shanmugam's user avatar
0 votes
1 answer
23 views

Appending and Inserting with Dask Arrays giving mismatch between chunks and shape

I have a Dask array with shape (1001,256,1,256) (data over 1001 timesteps with len(x)=len(z)=256 and len(y)=1). I need to pad the x-dimension with arrays of shape (1001,2,1,256), which I'm attempting ...
Dave's user avatar
  • 310
0 votes
2 answers
88 views

python calculation of tuples of two for big n

I'm trying to calculate all possible combinations of pairs in tuples starting from 0 to the length of my file (62977) so i get this result as a single DASK series\data frame with no duplicates: (0,1), ...
Mika Bell's user avatar

1
2 3 4 5
90