As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [multiprocessing]

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Using OpenCV and multiprocessing, how can I feed information from a variable in one Python script to another?

I am trying to have 2 scripts running in parallel and feeding one with the other. First I trained a model to decode different gestures. I followed the tutorial right here: https://www.youtube.com/...
simthi's user avatar
  • 1
0 votes
0 answers
34 views

how to get the number of the cpu cores that to which some tasks are assigned to process

i am working on a server with 128 processor. when i perform a prallelized operation as shown in the code section below,not all the cpu cores are utlitized, only some of the them as shown in the below ...
LetsamrIt's user avatar
  • 4,204
0 votes
0 answers
13 views

Using throttler and joblib for heavy jobs and getting "cannot unpack non-iterable function object"

I have a lot of heavy jobs which need to send throttled HTTP requests (2 requests/sec) and calculate the results based on the response values. I use throttler to control the request rate, and use ...
Xaree Lee's user avatar
  • 3,140
0 votes
1 answer
17 views

Multiprocess Sharing Static Data Between Process Jobs

Consider the following Python function to be parallelized, which utilizes an georeferenced ndarray (assembled from rioxarray) and a shapefile. This function uses both these datasets to generate map ...
TornadoEric's user avatar
-1 votes
0 answers
35 views

Python multiprocessing returns unpickable results

I am trying to time a function to factor primes and to optimize it, I want to use multiprocessing. The code below is what I have right now. I added a results class as seen in How to get the result of ...
Thomas Smeman's user avatar
0 votes
0 answers
14 views

Why does my speech recognition work every other time in the vosk model?

For practice , I decided to develop an api for loading a package of files in the format .zip. For me, it should have worked like this - a batch of files is loaded into fastapi and a message is sent to ...
Serega's user avatar
  • 156
0 votes
0 answers
9 views

How to speed up 3D Ising Model simulation with Python multiprocessing?

So I am trying to simulate the 3D Ising Model to generate magnetization values for a given temperature and lattice size using the Wolff Cluster Algorithm. I'm using Python to run all my codes and have ...
Praveen Murali's user avatar
1 vote
0 answers
14 views

Facing this error while run the code below for YOLOv8

I tried to create a training.py to run yolov8 on my computer but in the end it shows me this error: The code: ```from ultralytics import YOLO import torch # Create a new YOLO model from scratch model ...
Lim Chin Hong's user avatar
0 votes
0 answers
22 views

Which process is running on which cpu core? [duplicate]

There are 4 cpu cores in my pc. import os os.cpu_count() 4 I create 4 processes with Process module: vim mp.py import os,time from multiprocessing import Process def run_proc(name): print('child ...
showkey's user avatar
  • 399
3 votes
0 answers
35 views

A custom class of process or thread cannot be pickled

As you can see here, I can pickle both classes A and B: import pickle class A: pass class B(A): pass >>> pickle.dumps(A()) # Wroks >>> pickle.dumps(B()) # Wroks Also, I ...
Idan Hazan's user avatar
0 votes
1 answer
27 views

Bounded semaphore queue and processPoolExecutor

I recently chanced upon this neat trick by @noxdafox using a boundedSemaphore to hold a maximum queue length to limit the number of processes in the queue that is loaded into the processPoolExecutor. ...
markusyeo's user avatar
-1 votes
0 answers
28 views

My iPad ignores Multiprocessing commands?

I’ve been trying to learn python on my iPad - not a great idea, I’m aware. Most recently I have been looking at multiprocessing, and since there are no iPad specific videos, I’ve been using PC ...
Seb's user avatar
  • 1
0 votes
2 answers
75 views

Running multiple processes in each iteration of loop in Python

I have two functions func_1 and func_2. They both take an array of integers as input. I have a loop that creates arrays of length i in the ith iteration. I have two predefined lists list_1 and list_2 ...
Aniruddha's user avatar
  • 743
0 votes
0 answers
28 views

Xarray is very slow

Xarray is very slow. I needed to read 10 geotiff files (each of 10000 by 10000 size) and calculate mean across 10 images import glob import threading import numpy as np import rasterio, xarray, ...
mark's user avatar
  • 55
0 votes
1 answer
33 views

Convert joblib Parallel to multiprocessing Pool

I am currently using joblib's Parallel and it is working great but running into issues when I run in the server because the processes are not killed. I have a function that is called on a list of ...
codenoodles's user avatar
0 votes
0 answers
18 views

map_sync using an array as argument

I would like to map_sync a function using an array as argument. I obtain an error "TypeError: cannot pickle 'memoryview' object" when I run this simple code (I already have ipyparallel ...
rodiram's user avatar
1 vote
2 answers
51 views

How to execute parallel processing between dataframes?

I have two dictionaries dict1 and dict2, with the key value pairs being dictionary and dataframes. dict1 = {"A" : df1,"B" : df2,"C" : df3} dict2 = {"A" : df4,&...
Vinayak's user avatar
  • 43
3 votes
1 answer
71 views

Modifying a numpy array from multiple processes without locks

I have a big numpy.array shared across multiple processes (created using pool.map), and I want each process to modify a different part of the array. Since there is no 2 processes that can modify the ...
Neabfi's user avatar
  • 4,321
1 vote
2 answers
34 views

How can I make my class instance serializable for multiprocessing in Python?

I am currently working on a Mathematica-like library on Python called Mathics, and actually I want to make my program parallelizable with the multiprocessing module. But I have a problem, to make ...
Alexish's user avatar
  • 13
2 votes
2 answers
80 views

Is there a difference between Starlette/FastAPI Background Tasks and simply using multiprocessing in Python?

I am looking for different ways to queue up functions that will do things like copy files, scrape websites, and manipulate files (tasks that will take considerable time). I am using FastAPI as a ...
Wamy-Dev's user avatar
0 votes
1 answer
66 views

call concurrent futures from inside a function in another file

How can I use parallel computing sitting inside a function in another file that is being imported? Here as an example I created a file scratch_tests.py import concurrent.futures def g(): print(...
guyguyguy12345's user avatar
0 votes
1 answer
23 views

prevent AssertionError: daemonic processes are not allowed to have children

I am running stuff using Pool/multiprocessing. Whenever I call Pool again (nested) within one of the child processes of the main process, this error is raised? prevent multiprocessing gives ...
00__00__00's user avatar
  • 4,762
0 votes
1 answer
53 views

Overriden `Process.run` does not execute asynchronously

Having subclassed Process.run import multiprocessing as mp import time DELAY = 2 class NewProcess(mp.get_context().Process): def run(self) -> None: # add new kwarg to item[4] slot ...
vahvero's user avatar
  • 488
0 votes
0 answers
18 views

TypeError: 'Future' object is not iterable with Multiprocessing concurrent

I am trying to execute influxdb multiprocessing example: https://github.com/influxdata/influxdb-client-python/blob/master/examples/import_data_set_multiprocessing.py with concurrent.futures....
Anish's user avatar
  • 1,900
-1 votes
0 answers
61 views

How to scale up my kubernetecs pods for a CPU intensive python application?

I have a python application using fastapi and uvicorn that has some heavy cpu usage tasks. One of the task is excel to csv conversion using pandas. While running this task on a 200Mb excel file with 2,...
ayush sharma's user avatar
-1 votes
0 answers
36 views

Python how to call methods of a TeleBot object from another thread/process

I have a telegram bot and a web server, they work in different processes. How do I call the send_message method from the web server script to write to a user who is already registered in the bot and ...
SL_'s user avatar
  • 1
2 votes
1 answer
32 views

How to use properly Slurm sbatch and python Multiprocessing

I wan to run a code using multiprocessing in a server with slurm architecture. I want to limit the number of cpus available and that the code creates a child process for every of them. My code could ...
Simone Sartori's user avatar
0 votes
0 answers
21 views

pymodbus: get data from input registers paralleling using multiprocessing in python

I am using pymodbus library to get data from sensors using the below multiprocessing code def Get_Latest_Sensor_Data(tInput_Device_ID_List): sub_df = tInput_df[tInput_df["device_id&...
SM079's user avatar
  • 368
0 votes
0 answers
43 views

Python: Multiprocessing is_alive is False and exit_code is 0 and join did not worked

I am doing an experiment in multiprocess import multiprocessing import random, time import logging from logging.handlers import RotatingFileHandler logging.basicConfig( handlers=[...
vishak raj's user avatar
0 votes
1 answer
17 views

Sharing a large numpy array across python multiprocessing map

I have a script which is parallelized but requires reading from a large numpy array whose size might vary between different cases(can be 100GB+). The array is to be read-only. I have a class object(...
Bhavay's user avatar
  • 41
1 vote
1 answer
52 views

Multiprocessing Pool - too many file descriptors using Pytorch

Hello I'm trying to use the multiprocessing pool to try and play games simultaneously in a program I'm writing. Here is a code snipet of what i'm doing: self_play_pool = get_context("spawn")....
Guilherme Palma's user avatar
-1 votes
0 answers
16 views

python - build Windows exe fails with multiprocessing [closed]

I use pyinstaller to build Windows exe from python file. In my python file used multiprocessing.pool().map(). But fails. I have tried with nuikta. But fails too. Please help me how I can build or use ...
Cá Mập Cắn Cáp's user avatar
0 votes
0 answers
43 views

how to use Fully Sharded Data Parallel (FSDP) via Seq2SeqTrainer class of hugging face?

I have 2 GTX 1080 Ti GPUs(11G RAM each one) and i want to fine-tune openai/whisper-small model which one of the hugging face transformers models. Also, I want to use Fully Sharded Data Parallel(FSDP) ...
vafa knm's user avatar
0 votes
0 answers
14 views

Error while running multiprocessing Queue in Gramine: No such file or directory

I met an error while I running a program in Gramine, I hope someone can answer it for me. Hear are my files' detail. This is my mainfest file: loader.entrypoint = "file:{{ gramine.libos }}" ...
Yanmeng Smart's user avatar
0 votes
1 answer
21 views

How to create instance of classes using multiprocessing BaseManager

I have placed classes in a list like this: compare_ref_handler_list = [ (RefType.REF_SEE, RefSee), # reworked (RefType.MENUSELECTION, RefMenu), ... then I started the manager at the ...
Hoang Duy Tran's user avatar
1 vote
0 answers
27 views

Why using 8 processes is slower than using 4 while multiprocessing large data? [closed]

I have a function in my Python3 program that performs quick sorting on an array. The function takes in an entire array and num_processes to determine how many sub arrays are needed to be formed and, ...
edenfrey's user avatar
1 vote
0 answers
20 views

degrading performance with gunicorn

We have a dockerised python-flask-gunicorn app with below gunicorn config. The python app uses multiprocessing module within itself. There is also timeout = 10 defined to kill workers running a ...
Prateek Trivedi's user avatar
0 votes
1 answer
25 views

How to implement WSGI while using multiprocessing in flask?

Suppose I have video process handler in a function, I want implement true parallel processing using multiprocessing module instead of threading. So my code looks like this in general: def ...
Muhammad Ikhwan Perwira's user avatar
0 votes
1 answer
35 views

How to combine 2 processes to execute 1 class method?

I only recently started studying python and there was a need to create a phylogenetic tree, however, due to its large size and GIL, it was extremely long. I tried to make my own trial class. When ...
Nock's user avatar
  • 1
1 vote
0 answers
32 views

In python, why can't I terminate each of processes in a for loop, then block until if all of them are terminated in another for loop

In python, why can't I terminate each of processes in a for loop, then block until if all of them are terminated in another for loop. For example: for i in range(len(processes)): processes[i]....
Kazzit's user avatar
  • 41
-2 votes
0 answers
36 views

Python: how (in principle) to make a visualization without affecting performance too much? [duplicate]

My idea is the following: Suppose I have a general non-parallellized python computation. For example, a massive operation on large pandas dataframe or numpy array. The use of vectorized pandas/numpy ...
emixd's user avatar
  • 19
-4 votes
2 answers
42 views

How can I use multiprocessing.Pool.map to process a 100GB+ file with a generator in Python?

How to use multiprocessing.Pool.map with very big iterable? I need to process big file of data using multiprocessing. File size is 100GB+ so its not possible to read file in memory. So I created ...
tsu rugi's user avatar
0 votes
1 answer
47 views

Output missing in a duplicated ML Azure Studio notebook when using multiprocessing - why and how to solve?

I was trying to optimize my python code using multiprocessing in ML Azure Studio. The code is running fine in the initial notebook and i am getting the O/P as well. I was expecting it to be correct so ...
Aakash's user avatar
  • 11
-1 votes
0 answers
14 views

(Python3) Using Multiprocessing to Perform Quick Sorting on N Sub-Arrays

Context I have a function called quicksort(arr, left, right) that takes in an array of numbers and two indexes left and right. The algorithm would then perform an iterative, in-place quick sort of the ...
edenfrey's user avatar
2 votes
2 answers
43 views

Displaying total progress bar when spliting array in portions with multiprocesssing

I have a code where I am computing the average of an array whislt displaying a progress bar. Here is my code: import numpy as np from tqdm import tqdm import multiprocessing as mtp def split_list(...
nop nop's user avatar
  • 63
1 vote
0 answers
28 views

I don't understand how python uwsgi is multi-processed

I don't understand that when running with uwsgi multi-process, the id() value of the object generated by each thread is the same, and the threading.get_ident() value of each process is the same. I ran ...
쿨해머's user avatar
1 vote
0 answers
20 views

Sharing memory mapped numpy arrays across functions without segmentation faults

I am trying to share numpy ndarrays with zero memory copies across python processes, but I keep getting segmentation faults. The main problem is that as far as the main thread is inside the function ...
German Eizaguirre's user avatar
0 votes
1 answer
76 views

Can PyQt dialogs be shown in all threads or processes of multi-thread/process applications?

I have a file manager application written in Python3 using PyQt5 running under Linux. I want the ability to perform multiple concurrent file operations like copy and move. I have tried several ...
dave's user avatar
  • 33
0 votes
0 answers
25 views

Terminating process kills all results

Using multiprocessing to evaluate some models. Use a shared list to store the results. Run the eval, add the results to the shared list. If a eval process takes longer than x, then terminate the ...
My Imperial Lord Daft Vader's user avatar
0 votes
0 answers
17 views

reading from the same queue simultaneously with multiple subprocesses

I am trying to simultaneously run three different python processes that should communicate together in this order: process1: this is reading stuff from websocket and every 10 minutes passing it to ...
jou55's user avatar
  • 1

1
2 3 4 5
284