Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.
14,200
questions
0
votes
0
answers
8
views
Using OpenCV and multiprocessing, how can I feed information from a variable in one Python script to another?
I am trying to have 2 scripts running in parallel and feeding one with the other.
First I trained a model to decode different gestures. I followed the tutorial right here: https://www.youtube.com/...
0
votes
0
answers
34
views
how to get the number of the cpu cores that to which some tasks are assigned to process
i am working on a server with 128 processor. when i perform a prallelized operation as shown in the code section below,not all the cpu cores are utlitized, only some of the them as shown in the below ...
0
votes
0
answers
13
views
Using throttler and joblib for heavy jobs and getting "cannot unpack non-iterable function object"
I have a lot of heavy jobs which need to send throttled HTTP requests (2 requests/sec) and calculate the results based on the response values.
I use throttler to control the request rate, and use ...
0
votes
1
answer
17
views
Multiprocess Sharing Static Data Between Process Jobs
Consider the following Python function to be parallelized, which utilizes an georeferenced ndarray (assembled from rioxarray) and a shapefile. This function uses both these datasets to generate map ...
-1
votes
0
answers
35
views
Python multiprocessing returns unpickable results
I am trying to time a function to factor primes and to optimize it, I want to use multiprocessing. The code below is what I have right now. I added a results class as seen in How to get the result of ...
0
votes
0
answers
14
views
Why does my speech recognition work every other time in the vosk model?
For practice , I decided to develop an api for loading a package of files in the format .zip. For me, it should have worked like this - a batch of files is loaded into fastapi and a message is sent to ...
0
votes
0
answers
9
views
How to speed up 3D Ising Model simulation with Python multiprocessing?
So I am trying to simulate the 3D Ising Model to generate magnetization values for a given temperature and lattice size using the Wolff Cluster Algorithm. I'm using Python to run all my codes and have ...
1
vote
0
answers
14
views
Facing this error while run the code below for YOLOv8
I tried to create a training.py to run yolov8 on my computer but in the end it shows me this error:
The code:
```from ultralytics import YOLO
import torch
# Create a new YOLO model from scratch
model ...
0
votes
0
answers
22
views
Which process is running on which cpu core? [duplicate]
There are 4 cpu cores in my pc.
import os
os.cpu_count()
4
I create 4 processes with Process module:
vim mp.py
import os,time
from multiprocessing import Process
def run_proc(name):
print('child ...
3
votes
0
answers
35
views
A custom class of process or thread cannot be pickled
As you can see here, I can pickle both classes A and B:
import pickle
class A:
pass
class B(A):
pass
>>> pickle.dumps(A()) # Wroks
>>> pickle.dumps(B()) # Wroks
Also, I ...
0
votes
1
answer
27
views
Bounded semaphore queue and processPoolExecutor
I recently chanced upon this neat trick by @noxdafox using a boundedSemaphore to hold a maximum queue length to limit the number of processes in the queue that is loaded into the processPoolExecutor. ...
-1
votes
0
answers
28
views
My iPad ignores Multiprocessing commands?
I’ve been trying to learn python on my iPad - not a great idea, I’m aware. Most recently I have been looking at multiprocessing, and since there are no iPad specific videos, I’ve been using PC ...
0
votes
2
answers
75
views
Running multiple processes in each iteration of loop in Python
I have two functions func_1 and func_2. They both take an array of integers as input.
I have a loop that creates arrays of length i in the ith iteration.
I have two predefined lists list_1 and list_2 ...
0
votes
0
answers
28
views
Xarray is very slow
Xarray is very slow.
I needed to read 10 geotiff files (each of 10000 by 10000 size) and calculate mean across 10 images
import glob
import threading
import numpy as np
import rasterio, xarray, ...
0
votes
1
answer
33
views
Convert joblib Parallel to multiprocessing Pool
I am currently using joblib's Parallel and it is working great but running into issues when I run in the server because the processes are not killed.
I have a function that is called on a list of ...
0
votes
0
answers
18
views
map_sync using an array as argument
I would like to map_sync a function using an array as argument.
I obtain an error "TypeError: cannot pickle 'memoryview' object" when I run this simple code (I already have ipyparallel ...
1
vote
2
answers
51
views
How to execute parallel processing between dataframes?
I have two dictionaries dict1 and dict2, with the key value pairs being dictionary and dataframes.
dict1 = {"A" : df1,"B" : df2,"C" : df3}
dict2 = {"A" : df4,&...
3
votes
1
answer
71
views
Modifying a numpy array from multiple processes without locks
I have a big numpy.array shared across multiple processes (created using pool.map), and I want each process to modify a different part of the array. Since there is no 2 processes that can modify the ...
1
vote
2
answers
34
views
How can I make my class instance serializable for multiprocessing in Python?
I am currently working on a Mathematica-like library on Python called Mathics, and actually I want to make my program parallelizable with the multiprocessing module.
But I have a problem, to make ...
2
votes
2
answers
80
views
Is there a difference between Starlette/FastAPI Background Tasks and simply using multiprocessing in Python?
I am looking for different ways to queue up functions that will do things like copy files, scrape websites, and manipulate files (tasks that will take considerable time). I am using FastAPI as a ...
0
votes
1
answer
66
views
call concurrent futures from inside a function in another file
How can I use parallel computing sitting inside a function in another file that is being imported?
Here as an example I created a file scratch_tests.py
import concurrent.futures
def g():
print(...
0
votes
1
answer
23
views
prevent AssertionError: daemonic processes are not allowed to have children
I am running stuff using Pool/multiprocessing.
Whenever I call Pool again (nested) within one of the child processes of the main process, this error is raised?
prevent multiprocessing gives ...
0
votes
1
answer
53
views
Overriden `Process.run` does not execute asynchronously
Having subclassed Process.run
import multiprocessing as mp
import time
DELAY = 2
class NewProcess(mp.get_context().Process):
def run(self) -> None:
# add new kwarg to item[4] slot
...
0
votes
0
answers
18
views
TypeError: 'Future' object is not iterable with Multiprocessing concurrent
I am trying to execute influxdb multiprocessing example:
https://github.com/influxdata/influxdb-client-python/blob/master/examples/import_data_set_multiprocessing.py
with concurrent.futures....
-1
votes
0
answers
61
views
How to scale up my kubernetecs pods for a CPU intensive python application?
I have a python application using fastapi and uvicorn that has some heavy cpu usage tasks. One of the task is excel to csv conversion using pandas. While running this task on a 200Mb excel file with 2,...
-1
votes
0
answers
36
views
Python how to call methods of a TeleBot object from another thread/process
I have a telegram bot and a web server, they work in different processes. How do I call the send_message method from the web server script to write to a user who is already registered in the bot and ...
2
votes
1
answer
32
views
How to use properly Slurm sbatch and python Multiprocessing
I wan to run a code using multiprocessing in a server with slurm architecture.
I want to limit the number of cpus available and that the code creates a child process for every of them.
My code could ...
0
votes
0
answers
21
views
pymodbus: get data from input registers paralleling using multiprocessing in python
I am using pymodbus library to get data from sensors using the below multiprocessing code
def Get_Latest_Sensor_Data(tInput_Device_ID_List):
sub_df = tInput_df[tInput_df["device_id&...
0
votes
0
answers
43
views
Python: Multiprocessing is_alive is False and exit_code is 0 and join did not worked
I am doing an experiment in multiprocess
import multiprocessing
import random, time
import logging
from logging.handlers import RotatingFileHandler
logging.basicConfig(
handlers=[...
0
votes
1
answer
17
views
Sharing a large numpy array across python multiprocessing map
I have a script which is parallelized but requires reading from a large numpy array whose size might vary between different cases(can be 100GB+). The array is to be read-only. I have a class object(...
1
vote
1
answer
52
views
Multiprocessing Pool - too many file descriptors using Pytorch
Hello I'm trying to use the multiprocessing pool to try and play games simultaneously in a program I'm writing.
Here is a code snipet of what i'm doing:
self_play_pool = get_context("spawn")....
-1
votes
0
answers
16
views
python - build Windows exe fails with multiprocessing [closed]
I use pyinstaller to build Windows exe from python file. In my python file used multiprocessing.pool().map().
But fails.
I have tried with nuikta. But fails too.
Please help me how I can build or use ...
0
votes
0
answers
43
views
how to use Fully Sharded Data Parallel (FSDP) via Seq2SeqTrainer class of hugging face?
I have 2 GTX 1080 Ti GPUs(11G RAM each one) and i want to fine-tune openai/whisper-small model which one of the hugging face transformers models. Also, I want to use Fully Sharded Data Parallel(FSDP) ...
0
votes
0
answers
14
views
Error while running multiprocessing Queue in Gramine: No such file or directory
I met an error while I running a program in Gramine, I hope someone can answer it for me.
Hear are my files' detail.
This is my mainfest file:
loader.entrypoint = "file:{{ gramine.libos }}"
...
0
votes
1
answer
21
views
How to create instance of classes using multiprocessing BaseManager
I have placed classes in a list like this:
compare_ref_handler_list = [
(RefType.REF_SEE, RefSee), # reworked
(RefType.MENUSELECTION, RefMenu),
...
then I started the manager at the ...
1
vote
0
answers
27
views
Why using 8 processes is slower than using 4 while multiprocessing large data? [closed]
I have a function in my Python3 program that performs quick sorting on an array. The function takes in an entire array and num_processes to determine how many sub arrays are needed to be formed and, ...
1
vote
0
answers
20
views
degrading performance with gunicorn
We have a dockerised python-flask-gunicorn app with below gunicorn config.
The python app uses multiprocessing module within itself.
There is also timeout = 10 defined to kill workers running a ...
0
votes
1
answer
25
views
How to implement WSGI while using multiprocessing in flask?
Suppose I have video process handler in a function, I want implement true parallel processing using multiprocessing module instead of threading.
So my code looks like this in general:
def ...
0
votes
1
answer
35
views
How to combine 2 processes to execute 1 class method?
I only recently started studying python and there was a need to create a phylogenetic tree, however, due to its large size and GIL, it was extremely long.
I tried to make my own trial class. When ...
1
vote
0
answers
32
views
In python, why can't I terminate each of processes in a for loop, then block until if all of them are terminated in another for loop
In python, why can't I terminate each of processes in a for loop, then block until if all of them are terminated in another for loop.
For example:
for i in range(len(processes)):
processes[i]....
-2
votes
0
answers
36
views
Python: how (in principle) to make a visualization without affecting performance too much? [duplicate]
My idea is the following:
Suppose I have a general non-parallellized python computation. For example, a massive operation on large pandas dataframe or numpy array. The use of vectorized pandas/numpy ...
-4
votes
2
answers
42
views
How can I use multiprocessing.Pool.map to process a 100GB+ file with a generator in Python?
How to use multiprocessing.Pool.map with very big iterable?
I need to process big file of data using multiprocessing. File size is 100GB+ so its not possible to read file in memory. So I created ...
0
votes
1
answer
47
views
Output missing in a duplicated ML Azure Studio notebook when using multiprocessing - why and how to solve?
I was trying to optimize my python code using multiprocessing in ML Azure Studio. The code is running fine in the initial notebook and i am getting the O/P as well. I was expecting it to be correct so ...
-1
votes
0
answers
14
views
(Python3) Using Multiprocessing to Perform Quick Sorting on N Sub-Arrays
Context
I have a function called quicksort(arr, left, right) that takes in an array of numbers and two indexes left and right. The algorithm would then perform an iterative, in-place quick sort of the ...
2
votes
2
answers
43
views
Displaying total progress bar when spliting array in portions with multiprocesssing
I have a code where I am computing the average of an array whislt displaying a progress bar.
Here is my code:
import numpy as np
from tqdm import tqdm
import multiprocessing as mtp
def split_list(...
1
vote
0
answers
28
views
I don't understand how python uwsgi is multi-processed
I don't understand that when running with uwsgi multi-process, the id() value of the object generated by each thread is the same, and the threading.get_ident() value of each process is the same.
I ran ...
1
vote
0
answers
20
views
Sharing memory mapped numpy arrays across functions without segmentation faults
I am trying to share numpy ndarrays with zero memory copies across python processes, but I keep getting segmentation faults. The main problem is that as far as the main thread is inside the function ...
0
votes
1
answer
76
views
Can PyQt dialogs be shown in all threads or processes of multi-thread/process applications?
I have a file manager application written in Python3 using PyQt5 running under Linux. I want the ability to perform multiple concurrent file operations like copy and move.
I have tried several ...
0
votes
0
answers
25
views
Terminating process kills all results
Using multiprocessing to evaluate some models.
Use a shared list to store the results.
Run the eval, add the results to the shared list.
If a eval process takes longer than x, then terminate the ...
0
votes
0
answers
17
views
reading from the same queue simultaneously with multiple subprocesses
I am trying to simultaneously run three different python processes that should communicate together in this order:
process1: this is reading stuff from websocket and every 10 minutes passing it to ...