Questions tagged [ray]
Ray is a library for writing parallel and distributed Python applications. It scales from your laptop to a large cluster, has a simple yet flexible API, and provides high performance out of the box.
726
questions
0
votes
0
answers
8
views
ProcessPoolExecutor and ray
I have written some code that follows the following structure. Basically I am accessing some files from the disk and I am doing some image processing operations on them.
The program below makes a pool ...
0
votes
0
answers
8
views
Only 40% of CPU usage with Ray / modin.pandas
I am manipulating some data from dataframes(650MB) so I am using parallel processing in python.
The problem is that I thought that it would require CPU usage of around 80-90% but it is requiring only ...
-1
votes
0
answers
27
views
Ray Dataset loading HDFS: Failed to close HdfsReadableFile: IOError: HDFS CloseFile failed. Detail: [errno 255] Unknown error 255
I am using Ray Dataset read parquets on the HDFS cluster.
import ray
test_dataset = ray.data.read_parquet('hdfs://[email protected]:9000/path/to/file.parquet')
test_dataset.take(1)
When reading ...
0
votes
0
answers
44
views
Synchronize many ray.remote functions in python
I'm working on a project where multiple nodes (let's say 4, arranged in a grid) are engaged in parallel computations across a time span from tau = 0 to 2. Each node reads its own and its neighbors' ...
0
votes
0
answers
12
views
Checking if TorchTrainer is using the available GPUs
I am trying to train on a given dataset using distributed training in ray. I ran a sample code:
import os
import tempfile
import torch
from torch import nn
from torch.nn.parallel import ...
0
votes
0
answers
29
views
pandas value_counts in ray
I am pretty new to Ray and trying to use it for some data analysis operations on very large datasets
Lets say I have data in the following format
score_A
score_B
1
2
2
3
1
2
2
3
7
2
5
11
pd[&...
0
votes
1
answer
10
views
Ray tranformation with column dtype list<element: double not null>
My dataset schema is the following in ray:
Column Type
------ ----
x_1 list<element: double not null>
x_2 list<element: double not null>
y int32
Right now if I try to ...
0
votes
0
answers
55
views
How can I launch a ray server in a linux/amd64/x86_64 docker container running on a macOS arm64/M1/Apple Silicon - without crashing?
I have a ray server running successfully inside a docker (docker engine 24.0.6) [docker desktop: 4.25.2 (129061)] container on my M1 (Sonoma 14.1) as arm64.
On the M1, I need to create - and ...
0
votes
0
answers
31
views
What is the correct technique for incorporating access to large machine learning data sets in a Ray Tune Tuner() object?
I'm attempting to use Ray Tune 2.8.0 to optimize the hyperparameters of a pytorch neural network model--a fairly standard use case as far as I can understand. Pytorch is distributed with several ...
0
votes
0
answers
5
views
How to vectorize an RLlib MultiAgentEnv similar to Gym/Gymnasium's VectorEnv?
I have a multi-agent RL environment that's implemented as an RLlib MultiAgentEnv. I would like to do training and analysis with this environment but with my own custom algorithm code and without the ...
1
vote
0
answers
11
views
Ray tune and non-CPU-bound tasks
I like the interface and functionality of Ray Tune. However, I've observed a large per-task overhead when I use non-CPU-bound tasks (the actual compute is done on a 3rd party service). I understand ...
2
votes
0
answers
42
views
Attribute Error initialising Ray in Python
My python environment details are below. This is a conda virtual env that I created just to test ray core as on this Ray getting started tutorial. Initialising Ray throws the error: "...
0
votes
0
answers
21
views
Ray task is not cancelled
I am running task in Ray but I am unable to cancel it; the task is still running from Ray dashboard, noting that run_ray_task and cancel_ray_task are called in separate APIs and no errors. or ...
1
vote
0
answers
108
views
"protocol version mismatch -- is your shell clean?" when using docker with ray
I am getting the following error when using docker with ray:
Digest: sha256:2c0f86c6209e02d1f9e3d0a611ac04d73c3ed132b8b9319a28e2069f1503fb53
Status: Downloaded newer image for us-west1-docker.pkg.dev/...
0
votes
0
answers
32
views
Ray head node fails to connect when running ray up
I am trying to configure ray cluster with the following cluster.yaml configuration. For GCP, I am required to input the image. With 50gb images, the head and worker nodes start easily but with 10GB ...
1
vote
1
answer
29
views
PermissionError when ray.init()
I cloned a conda environment to new computer, and run a ray project, but face the error. I find this error occurs when ray.init(), the details codes and info are follows:
>>> import ray
>&...
0
votes
0
answers
25
views
Ray inferencing not happening in streaming way
I am using ray for inferencing.
scale = ActorPoolStrategy(size=100)
scale_gpu = ActorPoolStrategy(size=4)
preprocessed_ds = ds.map_batches(CPUPreprocess, compute=scale, num_gpus=0, batch_size=...
1
vote
1
answer
98
views
python ray out of memory (OOM) with RAY_memory_monitor_refresh_ms set to 0
I'm using Modin to use pandas code that can't fit in memory. It works very well locally with dataset (30GB) and my RAM(16GB).
Now I want to speed this up and decided to run a cluster on GCP with modin ...
0
votes
0
answers
43
views
Function Parameters in Customizing Models in Ray RLLib
How severe does this issue affect your experience of using Ray?
High: It blocks me to complete my task.
I started using Ray RLLib (version 2.7.1) to solve a RL problem with customized environments ...
0
votes
1
answer
36
views
Python Ray - explanation of await queue.get_async(block=True, timeout=None)
Ray.util.queue.Queue.get_async has block argument. Why async method blocks as async means return without blocking, in my understanding.
Looked for an explanation in the user guide but not found a clue ...
-1
votes
1
answer
86
views
Issue with Multi-Agent Environment Observation Space in RLlib
I'm working on a multi-agent environment with two agents (nodes), where each agent has to decide whether to transmit or not based on their observations of their own capacity and the capacity of the ...
0
votes
0
answers
38
views
Ray-tune log results in two folders
I using ray-tune to fine-tune parameter.
Based on the docs, ray-tune log the results of each trial to a sub-folder under a specified local dir, which defaults to ~/ray_results.
To change ray_results ...
0
votes
1
answer
47
views
Is it necessary to use object store in python ray.serve?
In python's ray.core, you can use an object store that can be shared by each Actor as follows.
import numpy as np
import ray
@ray.remote
def Actor(a):
return
a = np.zeros((5000, 5000))
...
0
votes
0
answers
95
views
Ray Tune: Load a checkpoints and continue to train PPO
I train a ppo in the following way, with my config, custom Env and action masking:
config = (
ppo.PPOConfig()
.environment(env=CustomEnv))
tuner = tune.Tuner(
"PPO",...
1
vote
1
answer
55
views
In Ray, does worker node system logs get streamed back to header node or only stay in worker node by default
I noticed system logs on worker node become unavailable when the node is gone, suspecting the log only stays in worker node. Is that correct? If so, how does the worker node becomes available on Ray ...
0
votes
0
answers
28
views
Python Ray Cluster, tell Node A to never take LAST Task in Job
I have a 3 node cluster for Ray.
Node A - 1 cpu
Node B - 1 cpu
Node C - 1 cpu
And 25 tasks. If all things were equal, then A,B,C runs roughly 8 tasks and the last one is picked up by whoever finishes ...
0
votes
0
answers
62
views
Can not use ray datasets map_batches function to update database (a class object obtained by langchain)?
I want to update the database (a class object obtained by langchain.FAISS) with new data use ray.
But I can not obtain the updated database even if it has been updated in the function.
This is a demo:
...
0
votes
0
answers
208
views
Ray for numpy multiprocesing
I am trying to use Ray for multiprocessing on CPU and GPU
Unfortunately, i cannot share the whole dataframe but here is a replica of my dataframe.
import pandas as pd
import numpy as np
issues_df=pd....
0
votes
0
answers
40
views
Ray data read does not seem to run in parallel
When I materialize this,
dataset = ray.data.read_csv(s3_bucket_name+'/'+s3_folder_path_prefix+'/',partition_filter=partition_filter,filesystem=s3,convert_options=convert_options)
it works but the read ...
1
vote
1
answer
68
views
TypeError: buffer is too small for requested array
I am using Ray 2.6.3. I would like to rename a column in a dataset.
I am able to run map properly without my data column of images. However, the buffer is too small for the array when I include data. ...
0
votes
0
answers
75
views
Transformers pipeline with ray does not work on gpu
I am following the documentation for offline batch inferencing using huggingface on gpu
https://docs.ray.io/en/latest/data/batch_inference.html#using-gpus-for-inference
I get a warning
You seem to be ...
0
votes
0
answers
216
views
Using ray on Databricks I got error ArrowInvalid: URI has empty scheme: '~/ray_results'
On a databricks notebook I'm running
%pip install jinja2 --upgrade
%pip install mlflow --upgrade
%pip install pycaret==3 catboost optuna lightgbm hyperopt tune-sklearn ray[tune]
%pip install pandas==1....
1
vote
0
answers
13
views
Using one Neural Network model inside of ray.pool
I am concerned about using Pool() from ray library. This is simplified structure of my question:
import torch
import multiprocessing as mp
from ray.util.multiprocessing import Pool
def acquire_data(...
0
votes
0
answers
44
views
Issue with parallel estimation of Transformer embeddings
I am attempting to compute BERT embeddings for a list of sentences using the Ray parallelization framework. The objective is to speed up this computation by running each sentence list (item contained ...
0
votes
0
answers
76
views
Scaling Pytorch training on a single-machine with multiple CPUs (no GPUs)
What is the best way to accelerate PyTorch training on a single machine with multiple CPUs (NO GPUs)?
We need to speed up training for a customer, because the training dataset grew substantially ...
1
vote
0
answers
38
views
Ray task in nested loop results in out of memory
I'm trying to run a ray task in nested loop.
Each loop takes dataA which has 5000 rows of list.
In ray task, it takes dataB and dataC that are 5000 x 256, 5000 x 768 respectively.
Whenever ray task ...
1
vote
1
answer
169
views
Parallelize DeepFace on multiple GPUs
I am trying to use the DeepFace python library to do face recognition and analysis on long videos: https://github.com/serengil/deepface.
Using the library out of the box, I am able to get desired ...
1
vote
0
answers
154
views
Error while running ray function - The task's local raylet died. Check raylet.out for more information
I am trying to run Ray from a docker container on Mac M1 (ubuntu:20.04, built via the command docker buildx build --platform linux/x86_64 -t dev_container -f Dockerfile.dev .)
ray.init() is working ...
0
votes
0
answers
65
views
Ray Tune Hyperparamter Optimization with XGBoost - TuneNoNextExecutorEventError| RayActorError
I simply would like to tune my XGBoost with Ray Tune locally. However, my "actor" always dies unexpectedly before the ray trail could even start.
I tried different code variants and also ...
0
votes
2
answers
133
views
Running Ray in docker
I'm trying to run ray in a docker image that looks like this:
FROM python:3.10.5-bullseye
RUN pip install -U "ray[default]==2.6.3"
...
CMD ["ray", "start", "--head&...
1
vote
0
answers
40
views
Error while testing keras_custom_model.py of ray rllib
I'm encountering a persistent issue while working with the Tune library in Ray RLLIB with Python on a Windows system. I'm attempting to run a test script from the library titled: custom_keras_model.py
...
0
votes
0
answers
16
views
How train model using tensorflow and rayTune then upload best model and its log to S3
I'm using raytune to do hyperparameter optimization with tensorflow model.
What I want to do is after training, upload model.h5 and its training log upto S3.
My code is something like this:
tuner = ...
0
votes
0
answers
88
views
How to use Ray Clusters to read a parquet and run a computation
I'm using Dask to read two parquet files, compute them, then merge them. However, I'd like to use Ray Clusters to achieve this, in hopes of speeding up the computation; how can I change the code to ...
1
vote
0
answers
172
views
Implement PPO agent using RAY on custom Gym environment
I've been interested in Reinforcement Learning for a while. I've made a simple PPO agent, using Stable Baselines 3, PyTorch and Gym to trade Bitcoin using Binance data. This was relatively easy and ...
1
vote
2
answers
158
views
Running Ray on top of AWS Batch multi-node?
I am interested in running Ray on AWS Batch multi-node. This is a pattern that hasn't been done before on Ray, and thus, there's no documentation on it. But, I'd really like to try it since Ray can be ...
0
votes
0
answers
34
views
Large Scale Hierarchical Agglomerative Clustering With Custom Distance Function/Similarity Matrix
I’m working on a project where I need to run hierarchical agglomerative clustering on between 1 million and 10 million data points. I also need to use a custom distance function (I cannot use ...
1
vote
1
answer
208
views
How to end episodes after 200 steps in Ray Tune (tune.run()) using a PPO model with torch
I'm using the following code to import a custom environement and then train on it:
from ray.tune.registry import register_env
import ray
from ray import air, tune
from ray.rllib.algorithms.ppo import ...
0
votes
0
answers
29
views
XGBoost RayTaskError
I am trying to do model training and inferencing in Databricks clusters using the Ray framework.
I have the following example with the xgboost-Ray example. I have generated ray dataset from parquet ...
0
votes
0
answers
64
views
How to configure Grafana dashboard uid for Ray dashboard?
The default uid is rayDefaultDashboard (e.g. https://github.com/ray-project/ray/blob/ray-2.4.0/dashboard/client/src/pages/metrics/Metrics.tsx#L201-L202), is that configurable?
E.g. I'll have two ...
0
votes
0
answers
82
views
How to use ray to efficiently run inference onnx run time?
I am running my inference scripts with multiprocessing currently which takes up a lot of memory with loading models with onnxruntime so I am trying to use ray run inference but getting issue with ...