Questions tagged [dask]
Dask is a parallel computing and data analytics library for Python. It supports dynamic task scheduling optimized for computation as well as big data collections.
4,494
questions
0
votes
1
answer
25
views
Dask dataframe condition to compare column doesn't working
I'm trying to compare the column names of a dask dataframe to then change the column data type, however my condition never is true:
column_name = "names"
print(f"Col Name: {column_name} ...
0
votes
0
answers
17
views
How to further partition/split an already partitioned dask dataframe
I have a very large DataFrame, that I partition based on values in one column "A" using the dask.DataFrame.set_index() method. Such N partitions are still too large to fit into memory when ...
0
votes
0
answers
10
views
User-defined types for groupby apply with p2p shuffling
It seems that p2p shuffling is not supported for a dask dataframe that contains columns of user-defined types. This is not a limitation for tasks shuffling though. Here’s an example:
import dask
...
0
votes
0
answers
22
views
Connection problems with dask scheduler
I've set up a kubernetes cluster with GKE and installed the dask-kubernetes-operator.
When i try to start the cluster like this
cluster: KubeCluster = KubeCluster(custom_cluster_spec="cluster....
0
votes
0
answers
23
views
Dask dataframes: read_excel file by file with panda
I'm processing multiple different types of files in different sizes with different columns. Files aren't too big (max 5MB and 20-30k rows) but panda was struggling with so decided to rewrite it with ...
0
votes
0
answers
19
views
I am trying to use geopandas to find polygons with are within 0.5m of each other using Dask. How do I multithread this operation?
so the current code I have can find intersecting cells, but Dask will not run it in parallel - is there a reason why, and is there a simpler and faster way to do this?
The error message is located at ...
0
votes
0
answers
29
views
Dask dashboard doesn't show any processing activity until after Xarray datasets are written out
I have a program that reads in several Xarray datasets, does some merging, reordering, concatenating to produce several merged Xarray datasets, and then writes them out to several files. I read in the ...
0
votes
0
answers
13
views
dask sshcluster results in: RuntimeError: Cluster failed to start: Worker failed to start
I try to get dask sshcluster (https://docs.dask.org/en/latest/deploying-ssh.html) up and running results in:
RuntimeError: Cluster failed to start: Worker failed to start
Python = = 3.11.6
Dask == ...
0
votes
0
answers
35
views
Error while trying to compute Dask dataframe from sql
I created a Dask dataframe using read_sql_query method and when I try to compute (some operation like pivoting, require computing), it raises an SQL error stating that it did not find the column I ...
0
votes
1
answer
33
views
Why aren't modifications made to a python module's variables propagating to new parallel processes?
I have an embarrassingly parallel list of tasks I want to perform. Currently I'm importing the configuration for these tasks as a module.
example one-line configuration.py:
result_folder = "...
0
votes
1
answer
33
views
Dask array operation on chunks with no return
I have a dask array of dimension (C, Y, X) (for instance, (100, 50000, 50000)).
I want to perform an operation on each chunk that will add some value to a small numpy array, that we will call x. Note ...
0
votes
1
answer
54
views
Resampling huge Pandas Dataframe throws ArrayMemoryError
At work, we have a machine, that is logging some sensor values (e.g. temperatures, pressures, speed, etc.). I have a csv file with these values with somewhat in between 6 and 7 million lines and 13 ...
0
votes
1
answer
18
views
dask read_csv is fast but dataframe operation is slow
I am trying to improve speed of read_csv() then later dataframe using pandas 2. I tried dask today and read_csv() is indeed really fast. But dataframe operation is slow. why is that? how to improve ...
1
vote
1
answer
17
views
AttributeError: module 'dask' has no attribute 'set_options' after installing dask with conda
I am using annaconda and I just installed dask using below
https://anaconda.org/conda-forge/dask
conda install -c conda-forge dask
I can use read_csv() and no error, however, I got this error . I am ...
1
vote
1
answer
33
views
dask import errors, dataframe/client - version conflicts with pandas?
Not all versions of dask.dataframe and pandas are compatible. This has been already addressed in this question
I have tried several combinations but in combinations with more recent dask versions, ...
0
votes
1
answer
21
views
Dask LocalCluster creating threads inside subprocesses
I'm new to dask and I seem to have some issues with something that should be fairly straight forward so I'm sure I'm missing something.
I have a program that supposed to poll messages from kafka, then ...
0
votes
0
answers
16
views
Stitching together irregular rasters into an XArray dataset that's too large for memory
I'm working on a project that involves many layers of raster data over a large area. This data is currently being stored in a set of raster files, each covering a small irregular polygon. I'd like ...
0
votes
0
answers
31
views
rasterio doesn't import to local dask cluster
I am running an analysis on a large number of rasters with Dask in a Jupyter Lab notebook on a Windows computer. When I run my analysis using Coiled (cloud cluster), the analysis runs fine. However, ...
0
votes
2
answers
45
views
dask: dtype mismatch in read_json when convert-string=False
When reading a JSON-lines file that contains complex fields with dask I am unable to get the correct data types.
The JSON-lines file looks like this
{"col1": "some text", "...
0
votes
0
answers
39
views
Dask DataFrame: write multiple CSV by column
I have a Dask DataFrame with a column "file". I want to write each line of the dataframe to a CSV whose path is given by the "file" column.
For instance, on the example below, the ...
0
votes
1
answer
32
views
Issue while using map_partitions in dask with a power function on pandas table with list elements
I am having the following code in python developed using the Dask framework:
# Create a Pandas DataFrame
df = pd.DataFrame({
'A': [[1], [2], [3], [4], [5]],
'B': [[6], [7], [8], [9], [10]]
})
...
-2
votes
0
answers
23
views
Unexpected argument flush in dask
I am running a code to analyze my data and I keep getting this error that I cant resolve yet after trying for several days. I ran this code previously with no issues. here is my code:
%%time
...
0
votes
0
answers
47
views
Large memory usage when calculating std dev over rolling window
I have a 2D array, and I want to calculate the mean and standard deviation over a spatial window of win_size pixels (~50 pixels or more). I only need to extract this for a subset of pixels whose ...
0
votes
0
answers
16
views
Migration Prefect1-2 and multithreading
Please, need your help.
I am seeking a way to migrate my quiet complexity Flow code from Prefect-1 to Prefect-2. Flow worked perfectly on the old Prefect version. But when I remade it to 2-nd version ...
0
votes
1
answer
32
views
How to persist a dask dataframe loaded with dask.dataframe.from_delayed
I have a large sharded dataset stored in a custom format that would benefit greatly from dask.dataframe.from_delayed
However, I'm seeing odd behavior when trying to persist the resulting dataframe:
...
0
votes
0
answers
47
views
Typing when passing xarray DataArray objects to numpy ufuncs
I have a function with type annotations that looks like the following:
import xarray as xr
import numpy as np
def compute_relative_azimuth(sat_azi: xr.DataArray, sun_azi: xr.DataArray) -> xr....
0
votes
1
answer
16
views
Dask efficient indexing of column-stored data as dask arrays
I'm trying to find an efficient way to index a dataset that is stored by columns in zarr arrays loaded with Dask (Python). I have to perform some operations on it, among which indexing all the columns ...
0
votes
0
answers
30
views
Dask client causing memory expection?
I´ve been migrating some of my codes from pandas do dask. Many of them are already working. Studing a little bit more about dask, I´ve faced it "Client()" feature for monitoring usage. Nice ...
0
votes
1
answer
52
views
How to use large table from exasol database for statistical analysis in Python?
I have a table with 36 millions rows and i need to run different statistical analysis (e.g. hypohtesis tests, distribution analysis etc.).
Since i get a memory error when i use the export_to_pandas ...
0
votes
1
answer
61
views
Dask dataframe -containing columns that are numpy arrays- conversion to parquet error
I have a dask dataframe which I am trying to convert to parquet files. This dataframe has columns which are numpy arrays, but column type is "object".
When I try to do this:
name_function = ...
0
votes
1
answer
29
views
close dask client and cluster when catching any errors or exceptions
I am writing a python function to do data processing with dask. I want to automatically close the dask cluster and client if any errors or exceptions raised. So I use the with ... as: statement. The ...
0
votes
0
answers
21
views
How to tackle Dask unmanaged memory in Windows OS when using delayed functions?
I have the below traditional Python function, without any array-type flavour, but which I need to run many times. Hence, I used Dask-parallelization using dask.delayed. However, I can see a gradual ...
0
votes
0
answers
36
views
How to turn a dask scalar of type boolean into a boolean expression
I have a long code with many data manipulations in which at the end of it I want to get a boolean expression from comparing two dask series, this is the final part of my code:
...
0
votes
1
answer
27
views
Dask from_delayed() causing high memory usage
I'm following the instructions from (https://docs.dask.org/en/stable/delayed-collections.html) to create a custom data loader for a Dask DataFrame, which is basically this:
import dask.dataframe as dd
...
0
votes
1
answer
34
views
Simple transformation in Dask data frame produces unexpected result
I have what looks to be a simple task: Compute the grouped cumulative ('running') minimum of Dask DataFrame column.
My data frame looks like this:
import pandas as pd
import numpy as np
import dask....
0
votes
0
answers
26
views
'TypeError: Categorical.ordered must be the same' when computing dask dataframe after pivot_table
I have a Dask DataFrame with the following structure:
Dask DataFrame Structure:
A B C
npartitions=25
...
0
votes
0
answers
26
views
xarray Can't read closed raster file ValueError by using Planetary Computer
I would like to use the OTCI dataset from the Planetary Computer Sentinel-3 Land dataset, by accessning it with the "stacstack" api, convert it to an xarray DataArray and do some further ...
0
votes
0
answers
29
views
Dask quantile over grouped data
Given a Dask dataframe with columns "eco_regions" and "ndvi", is there any way to group the data by the eco_regions and then compute the 90th percentile (0.9 quantile) of the ndvi ...
2
votes
1
answer
42
views
How to use Dask to parallelize up a `sel()` operation in xarray?
I have an array of values called speed, and I'm mapping it to another array of values of the same shape called power by looking up the nearest value in a lookup table speed_to_power_lut. This process ...
-1
votes
1
answer
41
views
How to specify correct dtype for column of lists when creating a dask dataframe from_pandas?
When creating a dask Dataframe with the from_pandas method, the formerly correct dtype object becomes a string[pyarrow].
import dask.dataframe as dd
import pandas as pd
df = pd.DataFrame(
{
...
0
votes
1
answer
60
views
I used Dask to read my 7GB CSV but now there is an error
The error says "NotImplementedError: dd.DataFrame.apply only supports axis=1 Try: df.apply(func, axis=1)"
Here is my line of code:
# Read the CSV file in using dask
import dask.dataframe as ...
0
votes
1
answer
21
views
Parallelize function over 3D array chunked with Dask
I have a dataset ds (dimension mid_date,y,x) that hosts a 3D numpy array (ds.v).
I have a function that I want to apply on each y,x cell of the array, which returns a vector reduced in dimension ...
0
votes
0
answers
117
views
Error about field having incompatible type when reading parquet with dask.dataframe.read_parquet
I am using jupyter/scipy-notebook, with dask version:
>>> dask.__version__
'2023.9.3'
When reading this parquet file, using dask.dataframe.read_parquet, I get an error.
mme = dask.dataframe....
0
votes
0
answers
13
views
Dask task stream finishing much faster than actual computation
I am attempting to run a process in parallel using dask.bag, but the process is taking longer than the task stream seems to suggest.
I am on dask version 2023.9.3
I am on a single machine
...
0
votes
2
answers
64
views
Regex Field Match and Replace in a network-Python
I have a large csv (+1000000 rows) that I need to do a regex search and replace function on. In short, I need to take two columns and find matches between them; and then use the matched rows to ...
0
votes
0
answers
28
views
dask.read_csv much slower than pandas.read_csv with 50k columns
I have a csv with 1k rows and 50k columns.
I takes 9s to calculate the min of all columns with pandas
panda_df = pd.read_csv(path, sep=",") # 8s
mins = panda_df.min(numeric_only=...
0
votes
0
answers
45
views
Convert series of lists into dataframe with multiple columns
How do I convert a Dask series that contains lists of varying length into a dataframe with multiple columns? In Pandas it's:
import pandas as pd
ps = pd.Series([["a0", "a1"], [&...
0
votes
0
answers
42
views
"Failed to deserialize" when using dask.dataframe.from_delayed on a collection of pandas DataFrames
Describe the issue:
I'm currently trying to read from a database in parallel using Dask and Pandas. I'm creating a collection of Dask delayed objects using pd.read_sql and then using Dask's ...
0
votes
1
answer
23
views
Appending and Inserting with Dask Arrays giving mismatch between chunks and shape
I have a Dask array with shape (1001,256,1,256) (data over 1001 timesteps with len(x)=len(z)=256 and len(y)=1).
I need to pad the x-dimension with arrays of shape (1001,2,1,256), which I'm attempting ...
0
votes
2
answers
88
views
python calculation of tuples of two for big n
I'm trying to calculate all possible combinations of pairs in tuples starting from 0 to the length of my file (62977) so i get this result as a single DASK series\data frame with no duplicates:
(0,1), ...