Questions tagged [joblib]
Joblib is a set of tools to provide lightweight pipelining in Python.
687
questions
-1
votes
0
answers
15
views
BrokenProcessPool error in google colab when trying to run cell with SequentialFeatureSelector
When I try to run a cell in colab I get a BrokenProcessPool error, which I believe is due to the SequentialFeatureSelector. The error occurs at the last line, which is sfs.fit(X,Y). I imported joblib ...
0
votes
1
answer
9
views
joblib parallel function for two function outputting two pandas df giving 'DataFrame' object is not callable error
There are two function :
func_1(arg1, arg2):
...
return df1. # Returns a pandas dataframe
func_2(arg3):
....
return df1 # Returns a pandas dataframe
My goal is to run those ...
0
votes
0
answers
13
views
joblib persistence across sessions/machines
Is joblib (https://joblib.readthedocs.io/en/latest/index.html) expected to be reliable across different machines, or ways of running functions, even different sessions on the same machine over time?
...
0
votes
0
answers
19
views
Sklearn with Joblib raises "AttributeError: 'NoneType' object has no attribute 'submit'"
I get the weirdest error when I use joblib (1.2.0) wit scikit-learn. Here is a MWE:
import numpy as np
from joblib import Parallel, delayed, parallel_backend
from sklearn.neighbors import ...
0
votes
0
answers
23
views
No module named 'sklearn' while running django server
I saved a model using joblib and was trying to run server using but it shows no module as sklearn even though it is downloaded in the environment, can anyone please help?
0
votes
1
answer
20
views
When predicting new dataset should I use scaler.fit_trasform(new_dataset) or scaler.transform(new_dataset)
final_poly_converter = PolynomialFeatures(degree=3,include_bias=False)
final_poly_features = final_poly_converter.fit_transform(X)
final_scaler = StandardScaler()
scaled_X = final_scaler.fit_transform(...
0
votes
0
answers
25
views
how to do with joblib that multiprocessing inside multiprocessing?
is it workable to do multiprocessing inside multiprocessing with joblib?
such as:
def func1():
Parallel(func2)
def func2():
Parallel(func3)
def func3():
Parallel(func4)
def func4():
do ...
0
votes
0
answers
27
views
AttributeError: Can't get attribute 'tokenizer' on <module '__main__' (built-in)>
I have a joblib file that I generated. This file (model) uses a locally defined function tokenizer(). I understand that this function has to be defined when using the saved joblib model. This is how ...
0
votes
1
answer
156
views
How to customize attrs field hash
I'd like to use a Numpy array as a field value while keeping my attrs class hashable. For that purpose, I found joblib's hash() function to be a good means of hashing Numpy arrays. Is there any ...
2
votes
0
answers
32
views
Joblib threading with sqlalchemy not running parallel
I am attempting to use joblib to run multiple inserts to my AWS Aurora serverless database. However, my joblib set up is evidently not running in parallel.
def insert_delay(chunk):
# time.sleep(...
0
votes
0
answers
23
views
How to run a batch of commands in parallel in Python (as opposed to run a single command)?
I've learned how to use the joblib package to run a single command in several independent threads as below ("learnIIR" is the function I want to parallelize):
from joblib import Parallel, ...
0
votes
0
answers
19
views
Uses of Joblib (python) to run a FOR loop in parallel
I'm looking to implement a Parallel structure in one of my code, if it's actually possible with the pythons limitations (GIL). The code is using a FEM (finite element modelling) structure and the ...
0
votes
1
answer
39
views
error in loading a custom class in sklearn pipeline using joblib
I am trying to deploy an sklearn pipeline using FastApi
so first i saved my pipeline in a job lib file.
My pipeline looks like this:
pipe = Pipeline([('encoder', MultiColumnLabelEncoder(columns =['sg',...
0
votes
0
answers
25
views
Using joblib.memory and joblib.parallel together
I have the following code (removing the unnecessary parts):
location='./cachedir'
memory=Memory(location,verbose=0)
def epstArray_cost(f_array,g_array,b):
<costly computation>
return ...
0
votes
0
answers
27
views
I cannot get parallel computing working in Python (joblib)
import time
from joblib import Parallel, delayed
def fun(i):
return i**2
start = time.time()
for i in list(range(0,10000000,1)):
print(fun(i))
end = time.time()
print(end - start)
start = ...
1
vote
1
answer
52
views
Confusion regarding joblib.dump()
One way to save sklearn models is to use joblib.dump(model,filename). I have a confusion regarding the filename argument. One way to run this function is through :
joblib.dump(model,"model.joblib&...
0
votes
1
answer
42
views
Why joblib is not recommended when saving keras model?
According to this keras documentation, pickle is not recommended to save keras mode, and since joblib.dump() and joblib.load() are based on the Python pickle serialization model, joblib is also not ...
0
votes
0
answers
31
views
Trouble importing sklearn while using joblib
I have a prediction program that predicts binary classifications using classifiers. When I try to run the program I get the same issues over and over again:
Traceback (most recent call last):
File &...
0
votes
0
answers
24
views
ProcessPoolExecutor with loky in Windows, one process is not terminated
In my quest to make a parallelised python program (written on Linux) truly platform independent, I was looking for python packages that would parallelise seamlessly in windows. So, I found joblib, ...
0
votes
0
answers
26
views
Adding a numpy array while using joblib
I am using joblib to do compute over a function that goes over 1000's of points. In this sample code, I have reduced it to 6 points (x_snap, y_snap). The output of code keeps appending to the number ...
0
votes
1
answer
48
views
How to effectively use parallelization with Ray in Python?
I'm trying to learn how to use the Ray API and comparing with my code for joblib. However, I don't know how to effectively use this (my machine has 16 CPU).
Am I doing something incorrectly? If not, ...
0
votes
1
answer
40
views
Catch sklearn joblib output to python logging
When using sklearn, I want to see the output. Therefore, I use verbose when available.
Generally, I want timestamps, process ids etc, so I use the python logging module when I can.
Getting sklearn ...
0
votes
0
answers
23
views
Why the objected saved by joblib cannot be loaded by pickle?
I was trying to load a .pkl file that generated using joblib.dump(). but I got error:
UnpicklingError: invalid load key, '\x00'.
Isn't that they use the same protocol?
import joblib
import pickle
...
0
votes
0
answers
20
views
joblib temporary files safe to delete?
I suddenly ran out of disk space for computing and found out that joblib temporary files take a lot of space and I'm wondering if it's safe to delete them?
And also: shouldn't they be deleted ...
1
vote
0
answers
94
views
Load joblib/pickle files from Databricks file system
I have a problem: I can't see to load objects from dbfs (data bricks file system) outside of spark (I can load the data fine with spark but not with pandas).
The objects we want to load are joblib and ...
0
votes
0
answers
16
views
joblib parallel processing breaks when running with unordered list in for loop
I am trying to run my time intensive analysis using joblib parallel, using the same Class_A.func but for 3 different dataframes A_df, B_df and C_df
res_a = Parallel(n_jobs=-1, backend='loky', ...
0
votes
1
answer
66
views
HDBSCAN : clustering , persistance and approximate_predict()
I want to cache my model results in order to make predictions without redoing the clustering.
I read that I can do that with memory parameter in HDBSCAN.
I did that instead because I wanted to save ...
0
votes
0
answers
14
views
how to avoid TerminatedWorkerError while paralellizing a function
I'm trying to run an optuna optimization script for many different categories in parallel. Below is a simplified version of what I'm trying to do.
import pandas as pd
import sqlalchemy as sa
import ...
0
votes
0
answers
13
views
Can't Load tensorflow model from pkl file as throwing custom object Error
can't able to the model while i can easily load normal models without callbacks
i can't able to load a model with call backs
early_stop = EarlyStopping(monitor="val_accuracy",
...
0
votes
1
answer
19
views
Use joblib to check whether an integer is a prime number or not. If it is, calculate its square; otherwise, append None to the returned value
import joblib
def is_prime(n):
for i in range(2, n):
if n % i == 0:
return False
return True
def square_if_prime(n):
if is_prime(n):
return n * n
return ...
0
votes
0
answers
73
views
How to read multiple large size excel files quickly using pandas and multiple worksheets as sperate dataframe using parallel process in python
I need to read large size of multiple excel files with each worksheet as a separate dataframes with faster way..
using below codes got Pandas DataFrame as a list, inside list having multiple ...
0
votes
0
answers
24
views
Issue in modifying a for loop using joblib
I have a sequential set of code which generates a tuple of values for different stocks, which is passed to a multiprocessing pool to apply technical indicators. Below is the sequential piece of code, ...
0
votes
0
answers
116
views
Using joblib.load() with data which was scaled
I did a very simple linear regression program for learning and I wanted to use streamline to deploy it.
My code looks a bit like this this:
#separate features and target variables
X = df['Number of ...
1
vote
1
answer
83
views
PicklingError : Can't pickle <function> <lambda> at 0x0000026C13C94430> : it's not found as __main__.<lambda>
I have used CountVectorizer() on following data
It was not considering single letter words so I used lambda in vectorizer
data = ['E', 'C', 'Employee', 'Child']
from sklearn.feature_extraction.text ...
0
votes
1
answer
45
views
How to control number of cores of a method
I have the following code:
from sklearn_extra.clusters import KMedoids
def _compute_medoids(df, k):
k_medoids = KMedoids(n_clusters=k, metric='precomputed', init='k-medoids++').fit(df)
...
0
votes
1
answer
381
views
Error: _pickle.PicklingError: Could not pickle the task to send it to the workers. NotImplementedError: object proxy must define __reduce_ex__()
I am using Python with joblib.
What could cause this error?
_pickle.PicklingError: Could not pickle the task to send it to the workers.
...
NotImplementedError: object proxy must define __reduce_ex__(...
0
votes
0
answers
97
views
access cache results in google colab
I am doing some clustering on google colab and want to cache the computation, since I think the computation history can only be accessed that way.
This is what I wrote, and when I run it, I don't see ...
0
votes
0
answers
40
views
Parallelizing a simple matrix assignment in Python with MPI
I have a rather simple parallelization question that I can't seem to work out. I had parallelized a simple matrix assignment using joblib in Python which worked nicely on my workstation, but now I ...
0
votes
0
answers
42
views
scikit-learn RandomizedSearchCV doesn't utilise all available cores
I'm using RandomizedSearchCV to optimize and evaluate several ML models. I'm trying to run the optimization process using joblib Parallel(). But, when running my code on an HPC using 16 cores, only ...
0
votes
0
answers
26
views
My joblib file is returning inaccurate output
I was making a Diabetes prediction model, and as I was developing it using Google Colab, the model was mostly returning accurate values. However, when I dumped my model into a Joblib file, it's now ...
0
votes
0
answers
21
views
Access variables from parallel job lib
I have set up a job lib for parallel computing, so far, I have been able to use it in computing several metrics. I intend to compute g_mean, after the roc_auc. However, I am unable to retrieve y_test ...
0
votes
1
answer
287
views
Shared-memory pandas data frame object in joblib.parallel
I'm using parallel function from joblib to parallelize a task. All processes take as input a pandas dataframe. In order to reduce the run-time memory used it is possible to sharing this dataframe? All ...
2
votes
1
answer
94
views
Pyinstaller created exe file cannot load Decision Tree model using joblib
I created an exe file of my large python script using the following command -
pyinstaller gui_final.py --onefile --hidden-import=sklearn --hidden-import=ipaddress --hidden-import=PIL --hidden-import=...
0
votes
0
answers
42
views
How to distribute proccesses on different cpu nodes, not only cpu cores in python using joblib?
I have 10 cpu nodes and each node has 72 cpu cores. I am running a code that reads huge files that need to be read in parallel to take less time. But the problem is that it goes to the memory limit. ...
0
votes
1
answer
60
views
What happens with workers when they are done with their task?
I have a task, which I aim to parallelize with the help of the joblib-library. The function is fairly slow when ran sequentially, therefore I tried using parallelization paradigms to speed up the ...
-1
votes
1
answer
40
views
Access File In Another Directory (Python)
I tried to access files with with open but the files itself located not in the same folder (since I want to access files in many different folders).
import os
path = r"C:\Users\M S I\Desktop\TA ...
0
votes
2
answers
333
views
No module named joblib Python
I'm trying to run this : https://github.com/HansiMcKlaus/AudioSpectrumVisualizer
And so I followed needed to be done to run it.
I pip install all the requirements, ffmpeg and try to run with the ...
0
votes
1
answer
23
views
Debug IO stall (state D) in parallelized python program, low CPU high memory
I have a Python program parallelized with joblib.Parallel,
however, as you can see in this top screenshot,
each process is using much less than 100% of the CPU, and the process state is "D", ...
0
votes
0
answers
95
views
Error when using joblib in python with undetected chromedriver
when i use (self.links is an array of strings)
Parallel(n_jobs=2)(delayed(self.buybysize)(link) for link in self.links)
with this function
def buybysize(self, link):
browser = self....
0
votes
0
answers
83
views
Using joblib to merge multiple Dataframes from list
I would like to merge multiple DataFrames stored in a List in to one DataFrame by using joblib / multiprocessing.
All DataFrames in list have the same columns.
df0 = pd.DataFrame({'name':['alice','...