Questions tagged [supercomputers]

Supercomputers belong to a class of highly specialised hardware infrastructures, where high number of machines are typically pre-organised and smart-linked together with specialised high-speed low-latency interconnects, so as to allow new forms of concurrent processing cooperations to be orchestrated. Having any such supercomputing infrastructure is not enough, it is important to also use system tools capable to harness the most of the available CPU-powers

supercomputers
Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Why can CPU memory be specified and allocated during instance creation but not GPU memory on the cloud?

On cloud computing platforms like AWS/Cloud/Slurm you can specify the amount of CPU memory that you want. But you can't do this for GPUs. For example H100 has 80 GB memory and you can't specify ...
JobHunter69's user avatar
  • 1,989
0 votes
1 answer
147 views

Failed to connect to the supercomputer platform using vscode ssh-remote

When I used vscode ssh-remote to connect to the Dawning 6000 supercomputer platform, the log file reported the following error: [2023-09-08 09:23:49] error This machine not meet Visual Studio Code ...
qinian_zovo's user avatar
0 votes
0 answers
60 views

Srun only launching one process

I am trying to run a job using slurm, but the srun is only launching 1 process, the job script I am using is below #!/bin/bash #SBATCH -N 1 #SBATCH -p RM #SBATCH -t 00:01:00 #SBATCH --ntasks-per-node=...
Sap BH's user avatar
  • 91
0 votes
1 answer
39 views

How to include aws credentials and configs while submitting PBS script jobs?

How do you load the aws credentials and config file in PBS scripts(.pbs) file? I want to submit a job that includes file transfer from a remote server to s3 buckets, but getting endpoint url error. ...
codecodecode's user avatar
0 votes
0 answers
72 views

How to upgrade a python library in a supercomputer

I am using Tinaroo (University of Queensland) super computer When I call to run my code using qsub 70my_01_140239.sh I get this error autosklearn.util.dependencies.IncorrectPackageVersionError: found ...
asmgx's user avatar
  • 7,783
0 votes
0 answers
128 views

why is Rpeak different from Rmax when measuring performance?

Rmax is maximum performance RPeak is theorotical maximum performance. but why can't supercomputers reach Rpeak. what causes the inefficency? an explanation to the cause of inefficency.
mTarifi4's user avatar
4 votes
1 answer
1k views

Slurm: how to use all cores available to the node?

I'm working with a large computing cluster with a SLURM workload manager that has four different subsections: we'll call them C1, C2, C3, and C4. Nodes in C1 and C2 have 28 cores, whereas those in C3 ...
Heatherosa's user avatar
0 votes
1 answer
268 views

Script is not working with high performance computer

I am using Tinaroo (a high-performance computer in University of Queensland) I built a demo python code (demo1.py), simple that creates a file that has the time of the execution of the code. import ...
asmgx's user avatar
  • 7,783
7 votes
2 answers
6k views

What is causing my random: "joblib.externals.loky.process_executor.TerminatedWorkerError" errors?

I'm making GIS-based data-analysis, where I calculate wide area nation wide prediction maps (e.g. weather maps etc.). Because my target area is very big (whole country) I am using supercomputers (...
jjepsuomi's user avatar
  • 4,313
1 vote
0 answers
323 views

SLURM squeue results - explanation of how users uses nodes

As a noob, I have access to supercomp with SLURM. The squeue command gives the list of used nodes for various jobs of different users. A small example is given below. Why do some users e.g. user1 (...
multipole's user avatar
  • 111
0 votes
0 answers
293 views

Adding HPC Cluster nodes to a Kubernetes env running on local VM/host

I would like to run Kubernetes on a local VM/host and add compute nodes allocated via slurm job allocation on the remote supercomputer. The compute nodes on the remote supercomputer are accessible on ...
Amit Ruhela's user avatar
0 votes
0 answers
427 views

python script on Google Cloud Platform still slow

thanks in advance for your patience - I'm not a programmer, but a researcher. I have a model worked up in a short python script that is computationally intensive. A time horizon input of 3 takes about ...
Steve Petersen's user avatar
2 votes
2 answers
105 views

How to setup amazon to use Grid computing with maple software that is in existing EC2?

I have one maple license in EC2 How to setup amazon to use Grid computing with maple software that is in existing EC2? I am not sure Grid computing whether is supercomputer. I just have a for loop ...
Ho Yeung Lee's user avatar
3 votes
1 answer
1k views

How to compute the diameter of 3D torus interconnect?

A 3D torus interconnect is a network topology having p^3 nodes where p > 2. A 3D torus is basically a 3D mesh with links connecting nodes on opposite faces (Am I right?). The bisection width ...
Shashank Kumar's user avatar
0 votes
0 answers
126 views

data exchange between multiple ranks with MPI_Bsend

I would like to ask a few questions (mostly question 2.) for the code below whose purpose is to send data to an arbitrary number of 'target' ranks then receive other data (of different length) from ...
Halbux's user avatar
  • 21
0 votes
0 answers
2k views

How to properly calculate CPU and GPU FLOPS performance?

Problem I'm trying to calculate CPU / GPU FLOPS performance but I'm not sure if I'm doing it correctly. Let's say we have: A Kaby Lake CPU (clock: 2.8 GHz, cores: 4, threads: 8) A Pascal GPU (clock: ...
AlekseyHoffman's user avatar
0 votes
1 answer
108 views

Is there any way to know which job will start next in qsub

In our institute (IISc Bangalore)Supercomputer ,we submit jobs using qsub. The jobs will start running according to the following- (1) Its wall time(Expected completion time) (2) Its position in ...
Debarshi Majumder's user avatar
1 vote
1 answer
451 views

Supercomputing: smaller number of nodes and more cpus/node vs. larger number of nodes and less cpus per node

On a supercomputer, you have a set of nodes, and for each nodes you have some amount of CPUs. Is it generally better if to use, say, 20 CPUS for 1 node, as opposed to 2 nodes with 10 CPUs each? In ...
24n8's user avatar
  • 2,086
0 votes
0 answers
299 views

Multiple processing using python to run scripts on a super computer

I am trying to run two python scripts as described below. This code is designed to download data from a specific URL of the file. years = ["2013","2014","2018","2019"] for year in years: code(year) ...
chintan thakrar's user avatar
3 votes
0 answers
92 views

How can I remove '\r' in a loop to read files in python

Does anyone know how to get rid of '\r'? I have to run 89 files in the supercomputer, I have done so far the two next scripts The first one is .sh to run it on shell $ID=$(sed "${PBS_ARRAYID}q;d" ...
Ale Lope's user avatar
1 vote
0 answers
150 views

Google Quantum supremacy - how are the 10k years assumed?

Last week, Google published a paper called Quantum supremacy using a programmable superconducting processor, which brags about: Our Sycamore processor takes about 200 seconds to sample one instance ...
gsamaras's user avatar
  • 72.7k
-1 votes
1 answer
176 views

Excel VBA script powered by supercomputers?

Is there a software/service/hardware available to run simple VBA scripts on a supercomputer (a remote cluster of CPUs)? I mean without installing the complex official HPC Excel extensions and spending ...
6diegodiego9's user avatar
0 votes
1 answer
180 views

how to make separate temp directories for each processes in a batch job

I have just started learning bioinformatics in my lab and I am a complete newbie. I am using a genome annotation tool called Kofamscan from NCBI and I am getting an error that could be due to the ...
Ryohei's user avatar
  • 29
0 votes
0 answers
32 views

How to make a clustercomputer testcode with python? (RPI)

I made a clustercomputer for a school project. Only i am having some problems with making a code that uses all eight Raspberry pi's. https://www.instructables.com/id/How-to-Make-a-Raspberry-Pi-...
Samuelvde's user avatar
2 votes
1 answer
336 views

Master SLURM Script that calls other SLURMS

I'm a cancer researcher and we were wondering if any of you knew how to make a SLURM script that calls other slurms. If the previous script finished successfully, continue to the next call. I.E ...
Bcheda's user avatar
  • 21
0 votes
2 answers
220 views

how to send a bash command to command line using python and execute next command without waiting for bash command to execute? [duplicate]

I want to execute a 3 separate bash commands on 3 separate core processors of a node of a supercomputer. I have a python script that sends a bash command via os.system to the command line, however it ...
user avatar
1 vote
0 answers
116 views

Slurm error: "slurmstepd: error: no task list created!"

I'm attempting to run a simple job on Slurm but am getting a cryptic error message: slurmstepd: error: no task list created! I've run thousands of other jobs identical to the job I'm running here (...
duhaime's user avatar
  • 26.6k
1 vote
1 answer
298 views

SLURM Embarrasingly parrallel submission taking too many resources

so I have the following submission script: #!/bin/bash # #SBATCH --job-name=P6 #SBATCH --output=P6.txt #SBATCH --partition=workq #SBATCH --ntasks=512 #SBATCH --time=18:00:00 #SBATCH --mem-per-cpu=...
Santiago Hernandez Orozco's user avatar
3 votes
1 answer
4k views

Slurm: how many times will failed jobs be --requeue'd

I have a Slurm job array for which the job file includes a --requeue directive. Here is the full job file: #!/bin/bash #SBATCH --job-name=catsss #SBATCH --output=logs/cats.log #SBATCH --array=1-10000 ...
duhaime's user avatar
  • 26.6k
0 votes
1 answer
2k views

Python: IOError 110 Connection timed out when reading from disk

I'm running a Python script on a Sun Grid Engine supercompute cluster that reads in a list of file ids, sends each to a worker process for analysis, and writes one output per input file to disk. The ...
duhaime's user avatar
  • 26.6k
3 votes
0 answers
81 views

What is the best way to set this working environment for my research group?

We recently got a supercomputer (I will call it the "cluster", it has 4 GPUs and 12-core processor with some decent storage and RAM) to our lab for machine learning research. A Linux distro (most ...
D_Serg's user avatar
  • 494
0 votes
1 answer
3k views

Performance of MPI_Reduce vs (MPI_Gather + Reduction on Root)

CRAY supercomputer using the MPICH2 library. Each node has 32 CPU's. I have a single float on N different MPI ranks, where each of these ranks is on a different node. I need to perform a reduction ...
wiowou's user avatar
  • 3
1 vote
2 answers
625 views

get available memory under SLURM using C++

I'm working in HPC environment and I'm using SLURM to submit my job to the queue. I'm writing my own memory caching mechanism and hence I want to know how much memory is available per node so that I ...
Anurag Peshne's user avatar
-1 votes
1 answer
41 views

Is it possible to change a job ID to something human-readable?

I'd like to send myself a text when a job is finished. I understand how to change the job name so that the .o and .e files have the appropriate name. But I'm not sure if there's a way to change the ...
Zack Vacanti-Mitchell's user avatar
1 vote
1 answer
177 views

Golem Task Settings : How can we configure diverse workloads and tasks

I have setup Golem Factory platform on my Mac Machine (MacOS 10.13.2). I could successfully setup the Golem Node. It is up and running. My Golem Wallet is showing a balance of 1000 GNT. Now I am ...
Gokul Alex's user avatar
1 vote
0 answers
355 views

Draw plot on a supercomputer using ipython

I want to plot a figure using python on a supercomputer. For example, I wrote a script plot.py: import numpy as np import matplotlib.pyplot as plt .... .... plt.plot(m) # m is a matrix with size (...
SYuan's user avatar
  • 171
-3 votes
1 answer
53 views

CPU performance with additional cores

How can I get faster processing speeds for a single thread by combining multiple CPU cores, like training a custom neural network (not tensorflow) on a Google Compute Engine n1-highmem-64 machine type ...
Jacob Edward's user avatar
0 votes
1 answer
190 views

parallel processing: class file has wrong version 49.0, should be 48.0

I run my code in supercomputer by used terminal. I have 2 kind of code: 1- sequential and parallel (shared memory) the sequential one it run correctly, but when I run the SM I have this error when ...
Njood Alassmi's user avatar
1 vote
2 answers
3k views

Why Torque qsub don't create output file?

I trying start task on cluster via Torque PBS with command qsub -o a.txt a.sh File a.sh contain single string: hostname After command qsub I make qstat command, that give next output: Job ID ...
r1d1's user avatar
  • 469
2 votes
0 answers
45 views

What addressing paradigms are being used for high performance computing?

What's the difference between "Active Global Address Space ( AGAS )" and "Partitioned Global Address Space ( PGAS )" used for high performance computing? It's not a specific question about ...
PreeJackie's user avatar
0 votes
0 answers
1k views

fortran: relocation truncated to fit, i can't create big arrays on server [duplicate]

I write a simple subroutine in fortran for a more complicated program. I need to read a file containing 3 columns of numbers (X_halo, Y_halo, Z_halo) and enter the 3 columns data into 3 arrays. In ...
Alessandro Peca's user avatar
2 votes
0 answers
446 views

Building a Roofline Model

I'm trying to build a roofline model for a node in a supercomputer that I'm running simulations on. The node has 2x Intel Xeon E5-2650 v2 (Ivy Bridge) 8 core 2.6 GHz processors (16 cores per node), ...
JC2188's user avatar
  • 337
3 votes
2 answers
3k views

When using qsub to submit jobs, how can I include my locally installed python packages?

I have an account on a supercomputing cluster where I've installed some packages using e.g. "pip install --user keras". When using qsub to submit jobs to the queue, I try to make sure the system can ...
user1634426's user avatar
0 votes
1 answer
288 views

Loop through all files in a directory simultaneously

I have a a loop within a loop The outer loop just goes through each file in a directory and runs the inner loop Can I select multiple files and have the inner loop running on these files ...
Lucas's user avatar
  • 65
0 votes
0 answers
311 views

How to run normal program in mpich clusters?

I'm making a cluster network(kind of super computer, but 'super fast') with my ubuntu installed computers, to run mprime of GIMPS Project. I used this tutorial. However, I don't know how to run mprime ...
Hyensoo Choi's user avatar
0 votes
1 answer
516 views

How do I save output from a large simulation in R? (multiple nodes, safe access)

I am doing a large simulation for a research project--simulating 1,000 football seasons and analyzing the results. As the seasons will be spread across multiple nodes, I need an easy way to save my ...
jntrcs's user avatar
  • 567
4 votes
1 answer
10k views

Weird "Stale file handle, errno=116" on remote cluster after dozens of hours running

I'm now running a simulation code called CMAQ on a remote cluster. I first ran a benchmark test in serial to see the performance of the software. However, the job always runs for dozens of hours and ...
Shangxin's user avatar
1 vote
1 answer
2k views

Supercomputer: Dead simple example of a program to run in supercomputer

I am learning how to use supercomputers to make the good use of resources. Let's say I have a python script, that will create a text file with given random number. myfile.py # Imports import random,os ...
BhishanPoudel's user avatar
2 votes
1 answer
171 views

Issues with supercomputer submission scripts to run NetLogo

I'm having some difficulty getting the supercomputer to run my netlogo model. I'm attempting to follow the advice given here and here. But no luck. I think it's not finding some of the needed jars? ...
Andrew Yoak's user avatar
0 votes
0 answers
31 views

Submitting jobs in parallel

I want to analyze data from different files (let's say 10) which are not related to each other. I want to know what is the difference between submitting the jobs in parallel or opening 10 terminals ...
Silviu's user avatar
  • 709