Questions tagged [cluster-computing]
A computer cluster is a set of connected systems that work together so that in many respects they can be viewed as a single system.
5,531
questions
1
vote
0
answers
12
views
What does the Aedes packages "aedes-persistence" and "mqemitter" do?
I am using the aedes MQTT broker. I read on this page that to use clustering, you need to mqemitter and aedes-persistence. But I am not sure what those packages do. My guess is that aedes-persistence ...
0
votes
0
answers
40
views
Inconsistent PCA Results with Different Multiprocessing Settings in sklearn
I'm encountering an issue with PCA in sklearn while using multiprocessing. Specifically, the reconstruction error in PCA varies significantly based on the number of processes set in Pool. For instance,...
0
votes
0
answers
14
views
Regarding the consumption of Kafka under the multi-process model of Node.js cluster
I encountered a problem. I used the node.js cluster model to register kafka consumers, but the multi-process did not work. Instead, only one process was continuously consuming. What is the reason for ...
1
vote
0
answers
26
views
R: Why do I get a Negative CH Index?
So, I'm trying to analyse my data following this tutorial :
https://enterotype.embl.de/enterotypes.html
At some point I need to determine the optimal number of clusters in my data calculating the CH ...
0
votes
1
answer
26
views
Run eclipse-mosquitto MQTT broker docker image and custom paho-mqtt Python clients inside minikube Kubernetes cluster
I want to run an MQTT broker and two clients inside a Kubernetes cluster and be able to send messages back and forth.
Therefore I wanted to use the eclipse-mosquitto Docker image and two custom ...
0
votes
0
answers
18
views
How to use TLS for secure communication in clusters ?
I'm implementing secure communication between clusters. However, there is very little information on this. It is only known that TLS can be used as a protocol for encrypted communication. But how do ...
0
votes
0
answers
19
views
How to further partition/split an already partitioned dask dataframe
I have a very large DataFrame, that I partition based on values in one column "A" using the dask.DataFrame.set_index() method. Such N partitions are still too large to fit into memory when ...
0
votes
1
answer
63
views
Incorrect Hostname of pods in same stateful set in an AKS cluster
I am at my wits' end here. I have been trying to solve this issue that all of a sudden started happening with my recent deployments to AKS to no solution. I have gone through a lot of different ...
0
votes
0
answers
13
views
Getting com.hazelcast.core.OperationTimeoutException
We are using Hazelcast as embedded in our spring boot application and our application is hosted on 12 tomcat servers forming a single cluster.
com.hazelcast.spi.properties.ClusterProperty....
1
vote
1
answer
34
views
SLURM/NFS cluster shared filesystem locked permissions on some users
Background
I used slurm to set up a 3-node cluster, one control node and 3 compute nodes (the control node doubles as a compute node). All of the nodes are on linux mint and they all communicate ...
-1
votes
0
answers
27
views
Flutter Cluster Manager per region
In my flutter app I use map service. I have places around let's say Europe. I'm using Cluster Manager for flutter in order to group them. When I zoom out a lot (in country level) they all merged in ...
0
votes
0
answers
8
views
Can Apache Mesos Be Used to Cluster Servers Across Departments Without Altering Application Code?
I'm working on an infrastructure project where our organization has several physical servers across different departments, each running its own set of applications and websites. I'm considering using ...
2
votes
1
answer
38
views
Problem running single-node job on computing cluster in conda environment for all processors in R
I am trying to run a single-node job on my university computing cluster, submitted via qsub, within a conda environment "myenv". As soon as I start my parallelization within my R script, the ...
0
votes
0
answers
27
views
Cluster configuration test failed in Vivado
l open settings options, and use cluster Configuration.
When clicking test Configuration button frequently, it errors occasionally.
Prompting content as follows:
Job <502467>: job has already ...
0
votes
0
answers
22
views
Slurm Error: Error in saveRDS(fit, paste("outputs3/fit_", mod, ".rds", sep = "")) : error writing to connection
I have a batch script that I am trying to run. I've made some changes to the original script (which ran and saved successfully) so I wanted to rerun the models and save to a different output folder. I ...
0
votes
0
answers
20
views
When I make custom cluster marker on google map via swiftUI, create class CustomClusterRenderer: GMUDefaultClusterRenderer
To override the renderer GMUDefaultClusterRenderer use the following code:
class CustomClusterRenderer: GMUDefaultClusterRenderer {
override func renderer(_ renderer: GMUClusterRenderer, ...
0
votes
1
answer
25
views
TLS/SSL certification error when intalling pip packages over SSH HPC clutser in virtual enviroment
I'm trying to install python packages in a virtual venv over HPC cluster.
(metaTS_venv) [xxxuser@hpc metaTS]$ pip3 install pandas
I get the TLS/SSL error below. It works fine if I install on the main ...
0
votes
0
answers
8
views
How should I distribute cores for inter and intra parallelization in TensorFlow?
I have been trying to run my neural network training on a cluster computer system. I am given a single node with 128 cores to use. I wanted to run 4 trainings in parallel, each using 32 cores at a ...
-2
votes
0
answers
16
views
Cluster and Crop cancer segments from images of the dataset
The dataset is split into train,test,val and each one have 7 cancer subtypes. What i want to do is to cluster each image and save important segments that contains cancer images from the each image.
...
-1
votes
0
answers
19
views
Accessing a remote minikube from a local computer
I'm trying to follow this guide in order to set up a cluster that I can connect to from another local machine: https://faun.pub/accessing-a-remote-minikube-from-a-local-computer-fd6180dd66dd#:~:text=...
0
votes
0
answers
9
views
Changing the Active Directory on an existing install of Windows HPC Pack 2019
I have an existing Windows HPC Pack 2019 install that is currently only working with an Administrator login.
I want to change it to work with a newly set up Active Directory. Is there a way to do this,...
0
votes
1
answer
17
views
Percona XtraDB Cluster: Automatic Failover Between Nodes in Java Spring Boot Application
I've set up a Percona XtraDB Cluster with two nodes, and my Java Spring Boot application is currently communicating with node1. The challenge is that when node1 goes down, the application doesn't ...
1
vote
1
answer
38
views
How to hold jobs for a user so that a total cpu number isn't reached in slurm?
I am submitting a bunch of array jobs 4 sets of 5 with 8 cpus each, so 4x5x8=160 cpus total. I would like to keep the running amount to below 100>2x5x8+1x2x8 cpus total, because I need to let ...
0
votes
0
answers
15
views
Clustering Words into given Categories in R?
I have a vector of words and two predefined categories: cat1 = c("Money, Commercial, Facility") and cat2 = c("Debt, Liability, Loans"). I'm interested in clustering the words in ...
0
votes
1
answer
35
views
Snowflake - Warehouse Node Utilization data?
I vaguely recalling someone stating that there is at least an "estimated" node usage of a Warehouse in Snowflake.
Not a cluster. The nodes of a cluster. For instance a Large Warehouse in ...
2
votes
0
answers
61
views
Scala TrieMap.computeHash fails unexpectedly on Spark Cluster, but HashMap works with same code
I am trying to refactor some existing code to run fully in cluster-mode and have been running into an unexpecteded NullPointerException under Scala 2.12.10 when trying to use the TrieMap, see
TrieMap....
0
votes
0
answers
16
views
Wildfly-29 clustering mode Logout Not Working - Session is Still Valid
i am running 2 nodes of Wildfly 29 in clustering mode, all works fine but when i tried to logout in the app, logout not working, the session remains alive.
I think the problem is the session cache ...
0
votes
1
answer
27
views
Vert.x SharedData - can it reliably store an LRU?
I'm trying to store a list of things in the Vert.x cluster shared data, where the list if kind of like a queue, but things drop off of the queue based on a limit of number of elements - basically the ...
0
votes
1
answer
27
views
Redis streams and multi/exec(transaction) in Redis Cluster
I want to publish a message to the Redis stream as part of Redis transaction(multi/exec). In a clustered environment, all data that is part of a redis transaction must be in same shard, so the ...
0
votes
1
answer
37
views
Databricks cluster policy reference oracle jar
How can I set a Databricks Cluster policy to reference an oracle jar file? I'm creating a policy via JSON (In the UI I'm not allowed to view the Libraries tab while creating the policy). I've done ...
0
votes
0
answers
27
views
How do I connect to a cluster with remote-ssh on vscode?
I am trying to connect to a ovh cluster using the ssh connection. I can connect but when im connected, the server crash immediatly with this log errors :
[13:00:38.785] Failed to parse remote port ...
0
votes
0
answers
13
views
Spark cluster - Stuck at start-all.sh and stop-all.sh
I am trying to configure a spark cluster, where I have one driver and one executor. After having done all the necessary setup, I am trying to start my cluster using the scripts in .../spark/sbin, ...
0
votes
0
answers
18
views
Problem with Clusters in NodeJS app + Socket.io
I was planning to optimize my Node.js application to distribute the load across all CPU cores, as a Node.js application can only utilize a single core. However, by using clusters, we can make use of ...
0
votes
0
answers
29
views
Python optimisation library to use on Databricks Spark cluster
I am running a Apache Spark cluster on Azure Databricks, on which I want to run an optimisation algorithm in Python. Are there any optimisation libraries built to work specifically with Spark?
I have ...
0
votes
0
answers
27
views
Cassandra cluster reset the modified value to initial cassandra.yaml configuration value when restart docker container
I'm running docker container with cassandra image. But running on vm occurs changing bridged-ip. I want to modify cassandra.yaml configuration. I tried modify cassandra.yaml configuration with docker ...
0
votes
0
answers
16
views
Spark cluster with different relative path on Dockerfile both for local and remote reference
I forked a spark cluster work. On file below, step ADD build/workspace/ ${SHARED_WORKSPACE}/ works fine on building action master.yml.
https://github.com/trouchet/sparkles/blob/master/build/docker/...
0
votes
1
answer
33
views
failed to find version script file when linking a dynamic library by using bazel cluster
I was building a shared/dynamic library by using bazel. The rules in Build file are like below
cc_library(
name = "a",
srcs = ["a.cc"]
)
filegroup(
name = "vis_ld&...
0
votes
0
answers
9
views
Docker tomcat scale with loadbalancer
I would like to implements docker-compose for my tomcat application where I can scale all tomcat containers that I need.
Does anyone can share with any docker-compose.yml example in order to see how ...
0
votes
1
answer
50
views
Databricks Cluster Spark configuration
How to run code in a Cluster's Spark Configuration. I've tried several settings to get Python to set the default database. Idea is that users don't have to specify the database when writing SQL ...
1
vote
1
answer
87
views
Automatically shift rows with same spatial coordinates into a different cluster
I have a dataframe df:
import pandas as pd
data = {
"latitude": [49.5659508, 49.568089, 49.5686342, 49.5687609, 49.5695834, 49.5706579, 49.5711228, 49.5716422, 49.5717749, 49.5619579, 49....
0
votes
0
answers
39
views
Snakemake: Handling Preempted Jobs on Slurm: Issues with Job Status and Cleanup
This message follows up on the one posted two years ago (Snakemake: Job preemption can interrupt running jobs on clusters, how to make sure that the task is not considered as failed?). I no longer ...
0
votes
0
answers
20
views
How to set resource limits to accounts for each partition in accounting file in slurm
We have SLURM deployed on our cluster with several partitions (part_1, part_2, part_3). We have created several accounts in the accounting file and several users are part of each account. For each ...
0
votes
0
answers
17
views
Jupyter Lab Remote Tunnel Link
Is there a way to launch a Jupyter Lab server that is accessible through a direct tunnel link (NOT via SSH) in a similar way that is available in VSCode Remote Tunnels?
The use case is when running ...
0
votes
0
answers
28
views
Automatic authentication with VSCode Remote Tunnel on a cluster compute node
I came across this helpful post on VSCode Remote Tunnel on a cluster compute node.
Is there a way to also automate the authentication (similar to ssh-key) such that one doesn't need to follow the ...
0
votes
0
answers
54
views
Segmentation Fault When Sending Arrays Over a Certain Size with Open MPI
I am writing a program to run with an ifiniband and intel-based cluster using openmpi, pmix, and SLURM scheduling.
When I run my program on the cluster with an input matrix over 38x38 on each node, I ...
0
votes
2
answers
66
views
Dask Distributed - Stateful global parameters shared by worker methods
I am using Dask to setup a cluster. For now I am setting up both the scheduler and the workers on localhost.
cluster = SSHCluster(["localhost", "localhost"],
...
1
vote
1
answer
219
views
Move everything from a k8s cluster to a new one
Is it possible to move every deployment on a running k8s cluster to another running k8s cluster?
I don't need to move cluster configurations, just every deployment, its pods, secrets, services, ...
0
votes
0
answers
19
views
How to divide the indices to determine the ideal number of clusters in 3 groups?
How to divide the following indices to determine the ideal number of clusters in 3 groups: internal cohesion (intracluster similarity),
distance between cluster (intercluster dissimilarity) or both?
...
1
vote
1
answer
26
views
killing child process in Node.js
If the child process is caught in an infinite loop, the 'exit' event is never handled.
How do you force a worker to stop in this case?
"use strict";
(async () => {
const cluster = ...
0
votes
0
answers
18
views
How to color branches of dendrogram [duplicate]
I am trying to color the branches for a dendrogram plot in order to better visualize similar and dissimilar years for my counts.
This is my current code I have:
library(dendextend)
BBbray_root.cluster ...