Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.
13,016
questions
0
votes
0
answers
4
views
Cache coherence for a data race free programs
All examples I've ever seen on when cache coherence is relevant are code examples that are data races (two cores simultaneously write to the same memory location).
When it comes to memory consistency, ...
2
votes
1
answer
26
views
Join timeout in multiprocessing
I have a dummy example, I want to apply multiprocessing in it. Consider a scenario where you have a stream of numbers(which I call frame) incoming one by one. And I want to assign it to any single ...
0
votes
0
answers
5
views
Error in multiprocessing with BigQuery RowIterator: cannot pickle '_thread.RLock' object
I'm running a piece of code with multiprocessing to process data from a BigQuery RowIterator, and after some 400 iterations, I ran into this error:
File "/opt/project/scripts/first_run.py", ...
0
votes
0
answers
10
views
Multiprocessing BrokenPipeError
I have a following problem. I am running a parallel task. I am getting this error:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in ...
0
votes
0
answers
33
views
python multiprocessing getting empty result
This is the regular function without using multiprocessing
def runMonte(loanpool, structured_securities, tolerance, NSIM):
structured_securities.addTranche(0.8, 0.05, 0)
structured_securities....
0
votes
0
answers
9
views
doing Ping in parallel in python
Im building an APP for controlling a robot. so first of all I wanna build a GUI and a welcome page so that the user can give the IP-address of the robot and User,pass credentials. meanwhile I wanna ...
0
votes
0
answers
9
views
Keep more then one openned HTTP sessions
I wonder if it is possible by using python to establish more than one opened HTTP keep-alive session by one user simultaneously.
For example, could I use multiprocessing to create more sessions ...
0
votes
0
answers
18
views
How two processor will share same function
I have this function here that is factoring a number n. Right now I am using two processors to run the same function and as soon as one of the processors get a result the process ends.
I was reading ...
2
votes
1
answer
38
views
How to do a race between multiprocessors in python
I have a function that factor a number. It depends on some random condition.
So what I am trying to do it's to run multiple processors in this function and the processor that finds the factor first ...
0
votes
1
answer
26
views
Broadcast message to multiple Multiprocessing processes
I have a Multiprocessing application with a bunch of processes, I want the control thread to broadcast a message to multiple processes. If I used a pipe pair for each there will be a ton of pipes.
Is ...
0
votes
0
answers
24
views
How to successfully implement Multiprocessing Lock?
I have been following the example synchronization-between-processes from the multiprocessing document. The script is as follows:
from multiprocessing import Process, Lock
def f(l, i):
l.acquire()
...
0
votes
0
answers
32
views
Why use multithreading when multiprocessing is available in python? [duplicate]
Apart from the memory footprint which is more in case of multiprocessing is there any motivation to use multithreading? From what I know the GIL in cpython would allow only a single thread to be ...
0
votes
1
answer
41
views
Split work between writing output from .exe and reading the output
I have a simple .exe downloaded from the apple app store. It gives real-time updates on crypto prices and their percentage change. I am extracting percentage changes of bitcoin.
I am using a ...
1
vote
1
answer
22
views
Multiprocessing uproot approach
I just started looking into parallelizing the analysis of rootfiles, i.e. trees. I have worked with RDataFrames where implicit multiprocessing can be enabled with one line of code (EnableImplicitMT())....
0
votes
0
answers
12
views
Delegate task from Multiprocess to Sub Process in Python
Hi currently I am using python telepot to:
Listen to incoming message
Based on the message it will do a long operation to generate some image.
Problem: The message is being process 1 by 1.
So if 2 ...
0
votes
0
answers
13
views
TypeError: cannot pickle '_thread.lock' object , EOFError: Ran out of input
Trying to use multiprocessing for running a watchdog for folder as a seperate process.
The watchdog should start running as a seperate process so that other functions will begin to run and populate ...
0
votes
1
answer
22
views
Implementing multiprocessing for a nested loop calculation (python)
I am trying to implement multiprocessing to speed up a nested loop. Following is a psudo code of what happens:
result = []
z = []
a1 = array of values
a2 = array of values
for i in range(len(a1)):
...
-1
votes
0
answers
9
views
Make codes run at once for all users
Is it possible to listen to each users contract events from the database at once I've tried but it only works for one user..
Helps will be appreciated
Web3 python telebot
from web3 import Web3
import ...
0
votes
0
answers
29
views
scipy.fftpack.fft with multiprocessing, how to avoid performance losses?
I would like to use scipy.fftpack.fft (and rfft) inside a multiprocessing structure,
I have observed significant performances losses due to an apparent incompatibility between scipy.fftpack and ...
-1
votes
2
answers
43
views
Split list automatically for multiprocessing
I am learning multiprocessing in Python, and thinking of a problem. I want that for a shared list(nums = mp.Manager().list), is there any way that it automatically splits the list for all the ...
0
votes
0
answers
9
views
how to read from h5py in multiprocessing without errors
I have code like:
def get_df(path, key):
with h5py.File(path) as hdf:
df = pd.DataFrame(np.array(hdf[key]))
return df
def f(key):
df = get_df(path, key)
...transform df...
...
0
votes
0
answers
20
views
A case for multiprocessing?
Say I have a function that gives me a lot of data coming from a device when called. I want to accumulate this data in a memory buffer. When the buffer reaches an arbitrarily chosen length, another ...
0
votes
0
answers
15
views
Is there any way to increase the size during memory sharing between process in PyTorch
My current code is like this:
import torch
import torch.multiprocessing as mp
t = torch.zeros([10,10])
t.share_memory_()
processes = []
for i in range(3):
p = mp.Process(target=some_function, ...
0
votes
1
answer
16
views
Launch four POST requests in parallel
I'm starting with multiprocesses and would like to understand a little more. I would like to know if multiprocesses can be applied to my situation and if so how can I do it. I have a situation like ...
0
votes
1
answer
29
views
How to dynamically change self variables, parameters, args... in multiprocessing?
I don't know much of Python yet, but I'm trying to create an app that controls multiple streams of sound simultaneously (It has to do with binaural beats, noise and the brain). Given that I would like ...
1
vote
0
answers
17
views
Multiprocessing hangs the python kernel, doesn't work at all [duplicate]
I got a new computer, AMD Ryzen 5800 (8cores), Win10, latest (ana)conda.
But it seems that multiprocessing doesn't work at all
import multiprocessing
import numpy as np
def my_func(A):
return np....
0
votes
0
answers
10
views
Multiprocessing emcee not working in Spyder (Python 3.9.7)
I'm new to this so please forgive me if I forget to add enough information. I have an MCMC code that uses emcee and that works perfectly fine in my "normal" terminal, with the Python 3.5.2 ...
0
votes
0
answers
15
views
Multiprocessing OpenCV in Python
I have a simple Algorithm, I want to run it fast in parallel. The algo is.
while stream:
img = read_image()
pre_process_img = pre_process(img)
text = ocr(pre_process_img)
fine_text = ...
0
votes
0
answers
13
views
Swoole\Process pop in blocking and non-blocking mode
Documentation says, that Swoole\Process::pop can run in blocking and non-blocking mode:
If failed, it returns FALSE. Or it returns the data. In the blocking mode, if the queue is empty, the call of ...
-1
votes
0
answers
30
views
How to use user input from one function in another function using multiprocessing?
Im basically trying to run several functions at once. One detects user input constantly, so when the user presses enter the user can enter a new input. Another function contains several if statements ...
0
votes
0
answers
17
views
whats the difference between “&” and multiple terminals in Linux Multiprocessing [closed]
I am using Linux for multi-task processing. Specifically, calling ffmpeg multiple times to transcoding multi source videos.
Strangely, when I am using:
“
ffmpeg-cmds01 &
ffmpeg-cmds02 &
ffmpeg-...
0
votes
0
answers
27
views
Uncompressing file should use multithread or multiprocess?
I'm uncompressing xz files using lzma module in python
with lzma.LZMAFile(compressed_path, "rb") as in_f, open(output_path, "wb") as out_f:
shutil.copyfileobj(in_f, out_f)
I'm ...
0
votes
1
answer
27
views
sys.excepthook in multiprocessing.Process ignored?
Suppose we have the two files namely mymanger.py and mysub.py.
mymanager.py
import time
from multiprocessing import Process
import mysub # the process file
def main():
xprocess = Process(
...
0
votes
1
answer
44
views
How to terminate all subprocesses when the first one completes
How to terminate both processes simultaneously? Code below starts and awaits for both processes to end execution until it terminates them, I want to terminate both processes when one of them reaches ...
-2
votes
0
answers
22
views
How to add a click module to python 3
I have a port scanner script
And I want to create in a separate file a click or something called a script.
import socket
from datetime import datetime
from multiprocessing.dummy import Pool as ...
0
votes
0
answers
6
views
what is the param chunksize mean in python2 mutilprocessing pool map function?
in python2, mutilprocessing pool function map, i set param 'chunksize=2' ,but it does not work. i confuse what that is mean? code as follows:
from multiprocessing import Pool
import time
import os
...
0
votes
1
answer
24
views
Python multiprocessing is slower if more packages are imported
I noticed that my code ran slower if I imported some packages (even if didn't use them).
Example1:
import time
from multiprocessing import Process
def donothing(seconds):
pass
if __name__ == &...
0
votes
2
answers
33
views
Is there a concurrent.futures.wait equivalent for the multiprocessing library?
I have a multiprocessing.Pool of workers, and I'd like to block the main thread until one worker completes. If I use concurrent.futures.ProcessPoolExecutor I can use concurrent.futures.wait to wait ...
0
votes
1
answer
21
views
Why can't I join a child process created by ProcessPoolExecutor or Pool?
I want to manually join and close child processes from either multiprocessing.Pool or ProcessPoolExecutor. However whenever I try to join a child process created by either of those Pools the code ...
-1
votes
2
answers
24
views
How to share a list of multidimensional arrays between python processes?
I am trying to speed up my code by splitting the job among several python processes. In the single-threaded version of the code, I am looping through a code that accumulates the result in several ...
0
votes
0
answers
9
views
Multi-processing port still listen after termination - Python
I have a task where I need to run some python file (call it app.py)
that uploads a server (using flask). This is done in run_tests function.
Then, I want to query this
server for some test inputs that ...
0
votes
1
answer
17
views
Swoole process is hanging forever
I have the simplest Swoole code, which sleeps for a second and prints "Run task" message to the screen.
<?php
namespace Tests\Util;
use PHPUnit\Framework\TestCase;
class ...
0
votes
1
answer
54
views
How to share an instance variable between multiple processes
I have a class defined and used as follows, that includes a method called in the context of a multiprocessing pool:
from multiprocessing import Pool
class MyClass:
def __init__(self, big_object):
...
0
votes
0
answers
29
views
How do I run a time consuming calculation in the background using Python and FastAPI?
I've been trying to wrap my head around Python multiprocessing but I'm getting stuck. I thought I could solve my problems with the use of a Pool but not really sure how to keep it running between ...
0
votes
0
answers
28
views
How can I use pygame.Clock in a multiprocessing.Process call?
I'm an inexperienced python programmer, and for an interesting project I thought I'd try making a multiplayer LAN game (using pygame), with a server script and multiple client scripts connected ...
-1
votes
1
answer
36
views
Multiprocess inside a multiprocess
eg.
I have a controller script.
I have a worker script.
I have 50 python objects that have to be passed to the worker script.
I want them to run in parallel.
The worker script has its own ...
0
votes
0
answers
17
views
How to Override Values In-Place for List using MultiProcessing in Python 3?
I've been trying to Run the following Code:
from math import inf
import multiprocessing
dist = [[inf] * n for _ in range(n)]
n = 8
data = [i for i in range(n)]
def set_self_distance(data):
for ...
0
votes
1
answer
28
views
why python ProcessPoolExecutor is not using all CPU cores properly for CPU bound program?
I am trying to learn how can I use multi core system more efficiently for CPU bound program in python. For this I am running a CPU intensive program in 3 different ways to compare in my i3 processor.
...
0
votes
2
answers
24
views
Python Multiprocessing Parallelize Inner or Outer Loop
Let's say we have some operation like:
groups = ['A','B','C']
idx = [n for n in range(1000)]
for group in groups:
for i in idx:
# Compute something
where idx is much larger than groups.
...
0
votes
1
answer
16
views
Accessing multiprocessing manager dict from webserver
I've been trying to share a multiprocessing manager dict with a web server so far with little success. The dict contains information but somehow it is not propagated to the running web server. The ...