Questions tagged [tpu]

Use this tag for Tensor Processing Unit (TPU). TPU is an application-specific integrated circuit developed by Google.

Filter by
Sorted by
Tagged with
0 votes
1 answer
23 views

Google Cloud Platform TPU pod v3.32 RuntimeError: TPU cores on each host is not same

Using an example of TPU usage on a pod v3.32 with software "tpu-vm-tf-2.11.0-pod" is always giving the same error when initializing TF's TPUStrategy: RuntimeError: TPU cores on each host is ...
0 votes
0 answers
17 views

InternalError during model.fit() while running on Colab TPU

I am attempting to fine-tune an expertBERT model from TensorFlow Hub on the Sentiment140 TFDS in Google Colab. However, I encounter the following error: InternalError: RET_CHECK failure (third_party/...
-1 votes
0 answers
11 views

Is there a way to pass arguments inside strategy.run when passing strategy to replica when using TPU for training a CNN in python?

Is there a way to pass arguments (other than tensors) inside strategy.run when passing strategy to replica (calculating replica losses for example) when using TPU for training a CNN in python? An ...
0 votes
1 answer
29 views

Error: Model provided has model identifier 'ctyp', should be 'TFL3'

I created a tflite model in python, but now I need to compile the model for EdgeTPU. When I use !edgetpu_compiler modelname.tflite in google colab, I am still getting this: Edge TPU Compiler version ...
  • 38
1 vote
2 answers
140 views

Cannot setup TPU in Colab using any methods with Jax

I have searched high and low and tried to set up TPU with flax. This notebook is my closest attempt. The error message is very strange. /usr/local/lib/python3.9/dist-packages/jax/tools/colab_tpu.py in ...
  • 155
0 votes
0 answers
14 views

How to do Gradient Accumulation on TF2.x with XLA and Multiple Train Steps

I am trying to experiment with TF2.x to achieve the best possible TPU speed. In this case, I have modified a few examples and come up with my own implementation. However, as the code stands I am ...
  • 155
-1 votes
0 answers
10 views

Not super code savvy person here, I've been getting "cannot import name '_TPU_AVAILABLE'

Had to shorten the error which is "ImportError: cannot import name '_TPU_AVAILABLE' from 'pytorch_lightning.utilities' (/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/init.py)&...
0 votes
0 answers
18 views

Error When using TPU on Kaggle. Tensorflow

I am trying to run a kaggle notebook using TPU's. It works fine on CPU and GPU's. However, I get an error when training model. It is as below. Please have a look. Would anyone be able help me in this ...
0 votes
0 answers
119 views

Is there a way to rewrite the Tensorflow gather_nd operation to be compatible to run on Edge TPU

I need to run a Tensorflow model that uses a similar function as the answer given in this question on Edge TPU. However, the Edge TPU does not support the tf.gather_nd operation. Is there a way to ...
0 votes
2 answers
60 views

Utilizing hardware AI accelerators with PyTorch

I'm pretty new to StackOverflow, but also to using PyTorch. I'm an AI and CS major, and I'm working on a project involving processing video with ML models. I'm not going to get into the details ...
0 votes
1 answer
78 views

Preemptible TPU - Data Managment while persistant disks are not available

I have access to 1 Preemptible Cloud TPU v3-32, and I want to train my LM on it, however, since it is preemptible, I can't attach a persistent disk (read-write mode) to it as it is also mentioned in ...
  • 671
0 votes
0 answers
29 views

Using a tensorflow edgeTPU model with two input tensors

I have a Coral USB arcceculator and I want to run a neuronal style transfer model on it with python, but I dont know how to handle two input tensors. I don't know how to do that, because the "...
  • 1
0 votes
1 answer
55 views

How to use TPU to accelerate sentiment analysis using FinBert in Kaggle

I am trying to analyze the sentiment of earnings calls using FinBert. Since I am analysing more than 40,000 earnings calls, the computation of the sentiment scores takes more than a week. Because of ...
0 votes
0 answers
49 views

My traied model with tensorflow on transformer pipeline pop out error

I’m using this github text summarization and I have a problem. I have been struggling for two weeks and I could not figure that out. I'm using a notebook from this github repository: https://github....
  • 13
0 votes
0 answers
26 views

Encountering Out of range error while training on TPU

I have been following notebooks on Kaggle to train models on TPU. This and This where the former notebook shows to make TF Records and the latter one to train models on TPU with these TF Records. I am ...
0 votes
2 answers
79 views

Failed to connect to TensorFlow master: TPU worker may not be ready or TensorFlow master address is incorrect

I signed up for the Tensor Research Cloud (TRC) program for the third time in two years. Now I barely created a preemptible v3-8 TPU. Before that, I could efficiently allocate five non-preemptible v3-...
0 votes
2 answers
47 views

healthAgent unhealthy-maintenance shown in htop output

This entry is on htop output for a tpu VM instance. Does unhealthy-maintenance indicate any problem with the tpu-vm?
  • 1,205
-1 votes
1 answer
3k views

can't open file : [Errno 2] No such file or directory

seohyun3545@tpu-vm:~$ python3 tomato.py python3: can't open file '/home/seohyun3545/tomato.py': [Errno 2] No such file or directory I cannot run python on the cloud shell. Since I used pwd to check ...
0 votes
0 answers
105 views

Tensorflow error Invalid argument Compilation failure: Detected unsupported operations when trying to compile graph cluster_tpu_function

I am receiving an error when I train my Deep Learning network using TPU The error is related to the below part of my code but I don't know how to solve it. preprocessing.RandomCrop(28, 28) NB: the ...
  • 384
2 votes
0 answers
86 views

InvalidArgumentError after step 6 during my training on TPU at Kaggle

I am getting an error when I train my model with TPU on kaggle. I already looked around the internet but can get any solution. Note that it is working if I use GPU but I run OOM error. Here is the ...
  • 384
0 votes
1 answer
149 views

Running Pytorch on Cloud TPU VM on GCP gives INVALID_ARGUMENT: No matching devices found for '/job:localservice/replica:0/task:0/device:TPU_SYSTEM:0'

I created a TPU VM on GCP. I am following the documentation page on how to run a calculation on a Cloud TPU VM by using PyTorch I have set the XRT TPU device configuration in the VM with export ...
  • 21.8k
1 vote
1 answer
201 views

Malformed entry 11 in list file /etc/apt/sources.list on fresh TPU VM on GCP

I created a TPU VM on GCP. I logged in via ssh and want to install some software. But I get the following error: $ sudo apt-get update E: Malformed entry 11 in list file /etc/apt/sources.list (URI ...
  • 21.8k
0 votes
1 answer
202 views

Cannot import PyTorch/XLA for using TPU in CoLab with ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found

I want to use TPU to run a PyTorch program on Google Colab, and I installed PyTorch/XLA in my CoLab notebook by using this code: !pip install cloud-tpu-client==0.10 torch==1.12.0 https://storage....
0 votes
1 answer
66 views

tensorflow training freezes randomly on kaggle tpu

I'm using kaggle TPU to train a tensorflow CycleGAN model. Everything is fine after training starts, but training freezes randomly after a few models. RAM has not exploded during training according to ...
0 votes
0 answers
59 views

Problem when using kaggle notebook with TPU accelerator: TypeError: can't multiply sequence by non-int of type 'NoneType'

I'm trying to create my own version of a CNN that is capable to tell if two images belong to the same person (facematch) and that one of those images is really a picture of a person or not (liveness ...
0 votes
0 answers
6 views

Execute arbitrary Pytorch subgraph on a particular device

I'm trying to run a fairly large existing model on TPUs via pytorch/XLA. A couple spots in this model have a lot of dynamic shapes that trigger frequent recompiles and trash performance. I'd like to ...
1 vote
1 answer
125 views

How to use AutoML Library with IPU/TPU?

I want to use AutoML Library Autogluon with Paperspace IPU/Kaggle TPU instance for specification reasons (big RAM, big space, and fast training time). For IPU, when I try to fit the Autogluon ...
1 vote
2 answers
414 views

Execute function specifically on CPU in Jax

I have a function that will basically instantiate a huge array and do other things. I am running my code on TPUs so basically my memory is limited. How can I execute my function specifically on the ...
0 votes
0 answers
67 views

How can I run this code on Kaggle TPU? Runs fine with GPU. Huggingface Transformers - text generation inference - facebook opt

What changes do I need to make? It runs fine with Kaggle 2x GPU. Thank you. !pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117 !pip install ...
0 votes
1 answer
219 views

TPU not found on Google VM (jax version 0.2.16)

I'm running a TPU v3-8 VM on Google. On the VM, I installed jax with pip install "jax[tpu]==0.2.16" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html. Unfortunately, I'm ...
  • 665
6 votes
1 answer
126 views

Test jax.pmap before deploying on multi-device hardware

My question is fairly simple: I am coding on a single-device small laptop and I am using jax.pmap because my code will run on multiple TPUs. I would like to "fake" having multiple devices to ...
0 votes
1 answer
85 views

403 permission error when tpu-vm writing cloud bucket

When I run the train.py script from https://github.com/tensorflow/models/tree/master/official/nlp, I got a 403 permission error. python3 official/nlp/train.py --tpu=con-bert1 --experiment=bert/...
  • 33
5 votes
4 answers
2k views

Unable to import name '_TPU_AVAILABLE' from 'pytorch_lightning.utilities'

I'm trying to import aitexten package to work on a GPT-2 solution. But I'm encountering an error:ImportError: cannot import name '_TPU_AVAILABLE' from 'pytorch_lightning.utilities' (/usr/local/lib/...
0 votes
0 answers
114 views

How do you install JAX on a Google Coral TPU?

The command I would like to use is pip install jax[tpu] But this rollbacks the version indefinitely (notice how instead of just installing the latest version it tries and then downloads the next ...
  • 446
0 votes
1 answer
146 views

Failing to connect with TPU-vm cause by error "ssh: connect to host [TPU-vm ip] port 22: Network is unreachable"

I have tried to connect with TPU virtual machine on linux machine under the network of my company. This failed with such a error reported: SSH: Attempting to connect to worker 0... ssh: connect to ...
  • 11
0 votes
0 answers
13 views

coralpy detect image example crashes with image file

I'm working on a project using the Coral TPU for image detection. Using detect_image.py with the supplied demo model and demo image this works just fine. And I've used it with some other images that ...
  • 1
0 votes
0 answers
18 views

Debugging speed in Google TPU

I observed a strange phenomena when I use google TPU. When I debug in GPU, the speed is really fast. However on TPU, when I debug, it takes so much time on printing the variable I want to identify on '...
1 vote
0 answers
110 views

How to see VRAM of google tpu?

Is it possible to see the VRAM of google TPU being used? In case of using gpu, we can identify how much memory is being used using nvidia-smi or gpustat, but I don't know how to identity it in tpu. I ...
0 votes
1 answer
82 views

TensorFlow save model UnimplementedError

I was using Kaggle TPUs to train a TensorFlow model and saving it here is the model code with strategy.scope(): model = keras.Sequential([ keras.layers.Input(shape = [None, None, 3]), keras....
1 vote
1 answer
42 views

Map a list of list to TPU tensor

I have this code to deconstruct a list of list to different tensors. token_a_index, token_b_index, isNext, input_ids, segment_ids, masked_tokens, masked_pos = map(torch.LongTensor, zip(*batch)) If I ...
  • 339
1 vote
1 answer
60 views

TPU-VM gives access denied when accessing Bucket

I have a number of Google cloud TPU-VMs which need to write to a Bucket. Most of them work fine, but the one I created this morning is giving me Access Denied when I run gsutil -m cp test_file.txt gs:/...
0 votes
1 answer
161 views

Training model on TPU VM aborts with core dump

I am trying to train T5X on the Winograd schema challenge. When I run my training script, I receive the following error. 2022-08-21 17:27:01.141608: F ./tensorflow/core/tpu/tpu_library_init_fns.inc:...
0 votes
0 answers
45 views

RuntimeError: tf.distribute.Strategy.gather method requires cross-replica context, use get_replica_context().all_gather() instead

The following 2 methods are described which give the error RuntimeError: tf.distribute.Strategy.gather method requires cross-replica context, use get_replica_context().all_gather() instead I am new to ...
  • 21
0 votes
1 answer
42 views

Is it possible to run a model in a Coral Dev Board in a way in which it doesn't ocuppies the whole screen?

I'm trying to build a setup in which I run my models on the Coral Dev Board locally, meaning that I can only access the shell directly from the monitor I have attached to the Dev Board, but I found ...
0 votes
1 answer
81 views

capture_tpu_profile failed to connect to all address

I'm trying to run the command from Cloud TPU Tools to monitor job. capture_tpu_profile --tpu=[my-tpu-name] --monitoring_level=2 --tpu_zone=[my-tpu-zone] It yields the following error 2022-08-07 ...
2 votes
1 answer
395 views

BERT pre-training from scratch with tensorflow version 2.x

i used run_pretraining.py (https://github.com/google-research/bert/blob/master/run_pretraining.py) python script in tensorflow version 1.15.5 version before. I use Google cloud TPU, as well. Is it ...
  • 33
1 vote
0 answers
35 views

Why does TPU strategy.scope() change keras.model.fit()'s input data type?

TPU strategy.scope seems to modify the data type of x in keras.model.fit().For example, # Case 1 model = unet() model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy']) ...
  • 43
0 votes
1 answer
131 views

convert a TPU tensorflow=1.15 model for deployment on GPU

I am struggling to load a TF1.15 TPU trained model to a GPU. The model directory has following files: model.ckpt-best.data-00000-of-00001 model.ckpt-best.index model.ckpt-best.meta I see these ...
  • 12.4k
1 vote
0 answers
195 views

dataset from tf.data.Dataset.save(load) for GCS + TPU got NotFoundError Could not find metadata file. [Op:MakeIterator]

I tried this both on TF 2.8 and Nightly (2.10.0-dev20220616) on Colab. In 2.8, i used the experimental version of this API. I am trying this in the context of trying to train a huggingface bert model. ...
  • 3,461
2 votes
1 answer
2k views

Google USB Coral TPU installation - unable to install PyCoral library

Environment: Ubuntu 20.04, Coral TPU USB accelerator I'm following these instructions - I was able to install the TPU Runtime after some Googling around unsigned package errors. However, when I try to ...
  • 21

1
2 3 4 5
9