Questions tagged [distributed]

Multiple computers working together, using a network to communicate

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

How does mnesia sync disc_copies with other nodes / PCs?

I want to back up my app using Mnesia so if the main node fails it could be restarted on another PC. So far I succeeded in connecting the main node to other nodes and their Mnesia, but I can’t make ...
-1 votes
0 answers
13 views

Distributed FS for kubernetes [closed]

I want to setup a k8s multinode cluster in virtualbox for my home Lab, want to use distributed storage for Persistent volumes, so that pods in all worker nodes should be able to mount the volume ...
0 votes
0 answers
24 views

Massive flaw in raft algorithm

So the raft dissertation and paper say this is how to handle append entries: Receiver implementation: Reply false if term < currentTerm (§5.1) Reply false if log doesn’t contain an entry at ...
  • 1,259
0 votes
0 answers
9 views

Celonis connector in Kafka connect distributed mode

I am trying to connect to Celonis EMS using EmsSinkConnector using Kafka-Connect in distributed mode. I ran kafka in distributed mode using below command. nohup bin/connect-distributed.sh config/...
0 votes
1 answer
28 views

Unable to use the Golang otel http client to propagate B3 headers to downstream service

I'm trying to use the otel packages to do tracing header (b3) propagation. Unfortunately I'm unable to get this to work. For the purposes of explaining, I have created a project on github which ...
0 votes
1 answer
18 views

JMeter Slave is unable to use the CSV when the count of Slaves are increasing

The load is 1000Threads from each instances. When I'm trying with 1 Master & 3 Slaves the jmeter is able to correctly use the CSV for slave instances however when the slaves are increasing , the ...
  • 1
1 vote
1 answer
29 views

init / sync mnesia on distributed rebar3 app

I have a rebar3 app which starts an Mnesia DB. I want to backup this DB on another node, so I've copied the code to another computer and tried to sync the nodes according to this and this. ...
0 votes
0 answers
11 views

Python Chat Client Only Responds After Sending a Message

When two clients are connected and are communicating with each other, the client doesn't receive anything until the client sends something itself. It looks like this: A: Sends message to B "hi&...
0 votes
1 answer
27 views

JMeter 5.5 Distributed org.apache.jmeter.engine.RemoteJMeterEngineImpl_Stub does not define or inherit an implementation

Running JMeter 5.5 in distributed, master and slave mode, the following command replicates this error Command jmeter -f -Gup=3 -Gtime=1200 -Gthreads=15 -R10.104.60.246,10.104.60.7 -GData=source.csv -n ...
0 votes
1 answer
21 views

Issue: Error in rconfigure() method java.rmi.ConnectException: Connection refused to host: 192.xx;

I have an issue when i run jmeter distributed : Error in rconfigure() method java.rmi.ConnectException: Connection refused to host: 192.168.200.22; nested exception is: java.net.ConnectException: ...
0 votes
0 answers
45 views

I can't lock value of the redis with redlock. How to lock the value in the redis?

I'd like to lock value in the redis with redlock. But, It's not locked even though lock is done successfully. This is my test code. Pushed data is 'refId:1234' and lock it for 4 seconds. So, pop ...
0 votes
1 answer
45 views

How do I distribute datasets between multiple GPUs in Tensorflow 2?

I'm trying to understand how to use multiple gpus to train a model on data too large for the GPU memory. Using tf.distribute.MirroredStrategy seems to copy the full data set to each GPU. What I'm ...
0 votes
0 answers
24 views

Celery - assign session dynamicall/reusing connections

Breaking my head for few days for simple task(Thought it simple...not anymore): Main program sends hundreds of sql queries to fetch data from Multiple DBs . I thought Celery can be the right choice as ...
  • 334
0 votes
0 answers
24 views

Tensorflow Distributed Multi-Machine Training: Failing to connect to nodes other than self

If I only use nodes 2-4 in TF_CONFIG, the program just hangs, where if I have one set as the primary (node01) the script throws the error logs below. I believe node01 is delegated as chief due to ...
0 votes
0 answers
56 views

python locust pass custom arguments to workers

I need to run locust in distributed mode. I'd like to use custom arguments, and it is needed to add specific value of argument to each worker. Here is sample python code: Locust test runner "&...
0 votes
0 answers
83 views

Python socket.sendall throwing broken pipe error

I am building a centralized logger where nodes send messages to a log and these messages are sent using the python socket library. Here is the code on the node side s = socket.socket(socket....
0 votes
0 answers
50 views

Apache Beam : Expected 2D array, got 1D array instead in distributed kmeans

so i have this code : class distKmeans(beam.DoFn): #i will do an init function to add the kmeans parameters def __init__(self, n_clusters,rseed=2): self.n_clusters = n_clusters self.rseed =...
0 votes
0 answers
40 views

How we can run Multiple Tests in K6

Please assist me in running a number of test scripts in the K6 Distributed Environment. I have multiple test cases, I looked online for articles or tutorials on this, but I couldn’t discover anything. ...
0 votes
0 answers
63 views

Selenium Node is Unable to Connect to Distributed Selenium Grid

I am currently trying to connect a node (PC #1) to my distributed selenium grid (PC #2), but my node is unable to make the connection. (PC #1 is Windows 10, PC #2 is Windows 11) I am using the ...
2 votes
0 answers
78 views

DDP and cuda graph in pytorch

This is my code and I am currently running it on 4 gpus setup(rank, gpus) dataset = RandomDataset(input_shape, 80*batch_size, rank) dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=...
0 votes
3 answers
183 views

Is it possible to resize a node pool disk size on GKE with terraform whitout recreating the cluster?

Is it possible to resize a node pool disk size on GKE with Terraform without recreating (first delete, then create again with new settings) the cluster? I want to automate the migration of the node ...
0 votes
0 answers
41 views

Elsa Workflows - batch processing in a distributed environment

I am thinking about using ELSA workflows in this scenario: Executing computational heavy activities on special nodes/servers Creating big number of workflows or better workflow instances similar what ...
  • 1
0 votes
0 answers
58 views

How to set up MASTER_PORT and MASTER_ADDR in slurm

In torch’s official documentation that talks about DDP, it said to set it as the following: def setup(rank, world_size): os.environ['MASTER_ADDR'] = 'localhost' os.environ['MASTER_PORT'] = '...
0 votes
1 answer
117 views

How to spin up/down workers programmatically at run-time on Kubernetes based on new Redis queues and their load?

Suppose I want to implement this architecture deployed on Kubernetes cluster: Gateway Simple RESTful HTTP microservice accepting scraping tasks (URLs to scrape along with postback urls) Request ...
1 vote
3 answers
83 views

How to distribute tasks between servers where each task must be done by only one server?

Goal: There are X number backend servers. There are Y number of tasks. Each task must be done only by one server. The same task ran by two different servers should not happen. There are tasks which ...
  • 105
0 votes
0 answers
29 views

Using torch.distributed modules on AWS instance to parallelise model training by splitting model

I am wondering how to do model parallelism using pytorch's distributed modules. Basically what I want to do is the following - class LargeModel(nn.Module): def __init__(self, in_features, n_hid, ...
  • 15
2 votes
1 answer
49 views

How can I stop an active query on a postgres coordinator when the worker node has crashed

I have a postgres select query that can not be stopped using the standard pg_cancel/pg_terminate commands. Both commands return true, but do nothing. The query has access share locks on hundreds of ...
0 votes
1 answer
56 views

What do the entries in Lamport clocks representations represent?

I'm trying to understand an illustrative example of how Lamport's algorithm is applied. In the course that I'm taking, we were presented with two representations of the clocks within three [distant] ...
0 votes
0 answers
83 views

How to use multiple GPUs for training?

I am simply trying to understand how to format a config file to allow multiple GPUs/distributed training to take place via the "train" command. The only clear tutorial out there is seemingly ...
0 votes
1 answer
13 views

Unable to Run Query with Cascade after Version upgrade from 20.07.2 to 20.07.3

We are new to the dgraph database. After Upgrading the server we are unable to run the below query with the cascade option `query get_tenantlevel_data($tenantuid:string){ get_tenantlevel_data(func: ...
5 votes
1 answer
224 views

Parallel for loop in Julia

I am aware there are a multitude of questions about running parallel for loops in Julia, using @threads, @distributed, and other methods. I have tried to implement the solutions there with no luck. ...
0 votes
0 answers
12 views

Parallel Textpreprocessing with Pyro

I am using distributed Gensim for Topic Modeling of documents. However, the preprocessing part is not distributed. Therefore I would like to implement a distributed preprocessing of the text data with ...
  • 21
0 votes
1 answer
59 views

what is the key difference between multipaxos and basic paxos protocol

how is multipaxos different from basic paxos? How does ordering work in multipaxos? Can someone explain multi-paxos along with the diagram Tried out going through videos and research papers but ...
0 votes
1 answer
102 views

distributed torch data collision from all_gather (writing all_gather results to file "fixes" the problem)

Problem: a distributed process computes errors and returns them alongside float indices when errors are gathered from the separate ranks, there's collision happening on these indices so if the ...
  • 772
0 votes
0 answers
13 views

What is the difference of RPC and collective communication

As opposed to point-to-point communication, collectives allow for communication patterns across all processes in a group. What are the main difference of RPC and collective communication? How they ...
  • 81
0 votes
0 answers
70 views

How to use ZooKeeper to distribute work across a cluster of servers

I'm studying up for system design interviews and have run into this pattern in several different problems. Imagine I have a large volume of work that needs to be repeatedly processed at some cadence. ...
1 vote
1 answer
54 views

Why does Paxos ensure that consensus is reached and does not change?

I have been reading about single-decree Paxos (primarily looking at Paxos Made Simple) yet am confused about if a consensus among Acceptors is guaranteed to not change after being reached. According ...
  • 13
0 votes
1 answer
292 views

torch.distributed.barrier() added on all processes not working

import torch import os torch.distributed.init_process_group(backend="nccl") local_rank = int(os.environ["LOCAL_RANK"]) if local_rank >0: torch.distributed.barrier() print(...
0 votes
0 answers
29 views

Using Dart for desktop, distributed and cloud programming

I would like to use Dart for desktop, distributed and cloud programming. Would Google extend the Flutter framework to include libraries for GUI desktop, distributed and cloud programming, in the same ...
0 votes
1 answer
161 views

How to do a groupby of a Ray dataset using two keys?

Let’s say I want to groupby A and B and calc the sum of Sales? How should I go about it? import pandas as pd import ray ray.init() rdf = ray.data.from_pandas(pd.DataFrame({'A':[1,2,3],'B':[1,1,4],'...
0 votes
1 answer
116 views

Does optuna.integration.TorchDistributedTrial support multinode optimization?

Does integration.TorchDistributedTrial support multinode optimization? I'm using Optuna on a SLURM cluster. Suppose I would like to do a distributed hyperparameter optimization using two nodes with ...
  • 17
0 votes
1 answer
48 views

Gensim distributed Word2Vec & Doc2Vec

I would like to implement a distributed Topic Modeling Pipeline based on Gensim. Unfortunately, Gensim only supports distributed LSA and LDA. I read that Word2Vec and Doc2Vec tend to have better ...
  • 21
0 votes
0 answers
16 views

UCAN spec questions: nonce recommendation, fact example, prf proof, owner UCAN cache mechanisms, session content ID, SPKI limitations

I have a few questions about the UCAN Spec : the spec explains that UCAN is of the SPKI family. However the paper Capability Myths Demolished linked in the Intuition chapter mentions that SPKI are ...
0 votes
0 answers
72 views

Difference between Statement-based replication and Write-ahead log (WAL) shipping?

I am reading book DDIA(Designing Data-Intensive Applications) and found these two ways of implementing replication logs. They both seem to be maintaining a log of all 'write in'. Can anyone explain to ...
0 votes
0 answers
143 views

Loss reduction in tf.distributed.MirroredStrategy()

I'm confused regarding using distributed strategy and the correct way of reduction in loss functions. I implemented a U-Net using tf.distribute.MirroredStrategy(). Everything works fine using default ...
  • 1
0 votes
2 answers
98 views

Kubernetes Distributed testing - Jmeter won't start; sun.rmi.transport.tcp.TCPConnection close

My architecture: namespaces in kubernetes behind proxy, firewall and other switches. My master sends tasks to the slaves and when the script ends, it closes the slaves and the master. I have netpols, ...
1 vote
0 answers
279 views

Redis Keyspace notification not working for go-redis in cluster mode

I am using go-redis/v8 as a client library and its universal client to get/set etc. I am subscribing to the keyspace notifications like the one below. keyspace := fmt.Sprintf("__keyspace@%d__:%s&...
0 votes
1 answer
235 views

PyTorch Distributed SocketTimeout error on Windows

i've developed this little POC using pytorch distributed package: essentially a Trainer spawns N processes and orchestrate them using python Pipes (it could also be Queues). Normally it should send ...
1 vote
0 answers
53 views

Using SharedArray and pmap in Julia

I am thinking about using distributed computation in a problem I face. Suppose I have an index k which increases from 1 to 800 (for instance). And for each k, I have a pool p which has large size and ...

1
2 3 4 5
45