2

I'm currently studying how to use TensorFlow and can't seem to grasp my head around what a Variable exactly is. I've read other sources such as the official documentation, this blog post and some other questions on this community but they don't seem to be actually explaining what a TensorFlow Variable is and why we use them.

My understanding is that Variables in TensorFlow seem to have some properties or functionality that is unique to TensorFlow (e.g. when we run TensorFlow sessions), but I'm not quite sure if this is exactly the case. What would be stopping us from building neural networks simply using pure Python and NumPy Ndarrays?

2 Answers 2

2

What would be stopping us from building neural networks simply using pure Python and NumPy Ndarrays?

Nothing at all. Well, nothing except for time, effort and know-how.

Conceptually, a variable is just that: a variable. Just like anything you would use in another program.

Of course, you are correct to say:

that Variables in Tensorflow seem to have some properties or functionality that is unique to Tensorflow

As an example: a variable is generally (although certainly not always) "trainable". This means it represents more than just a placeholder for a number. Operations using that number are 'tracked' by tensorflow so that during backprop this number can be adjusted according to the gradient of the loss function and whatever optimizer we are using.

This doesn't really alter the concept.

2

Just to expand a bit on what Stewart_R already said. For the most part, when you use TensorFlow operations you are using tensors (tf.Tensor, that is). These are "immutable" in the sense that if I write:

c = tf.add(a, b)

Then the new tensor c is always going to be the result of adding a and b. Now, a and b can be different things each time (for example if they are placeholders for which you feed a value), but c is always the result of adding them. So, in each call to session.run, if everything else is the same, c is going to be the same, always.

Unlike with NumPy, you cannot do something like:

c[2] = 3

If you want to create a tensor that is "like c but with the index 2 changed to value 3" you have to literally create a new tensor like that (which is a common kind of issue). Do not confuse this, though, with doing something like this:

c = something_else()

Now c will contain a reference to the result of something_else(), but the original tensor resulting of tf.add(a, b) will still be there, in the TensorFlow graph (tf.Graph) (this works a bit different in eager mode, but let's leave that for another occasion).

There are however another kind of entities you also deal with sometimes, which are variables (tf.Variable). The key feature of variables is that they can hold a value that is kept between different calls to session.run. How this works is, you declare them with a type and a size, and then you have assignment operations. When you run an assignment operation in a session, the value gets fixed in the session until new assignment operations are executed. Variables require you to assign them a value before using them for the first time in a session, as when you create the session they are in an "empty" state. This is what the initialization is for.

Variables are typically (but not exclusively or necessarily) used to hold parameters of a trainable model. For example, if you train a neural network, you have a several "weights" to train. On each training step, you call session.run and then an optimization step is performed using one batch of examples. Obviously, you want the next step to progress from the previous one, so you need to keep the values of the parameters between calls to session.run.

The flip side is that variable values only exist within a session. When the session is closed, the values are gone. Therefore, you have all sorts of mechanisms to save variable values out of a session, like checkpoints, saved models, HDF5 Keras files, graph freezing...

About just using NumPy, well, like Stewart says, you can. The problem is, to train a neural network (or similar thing), you need to compute the gradients of the operations that you are doing, and NumPy does not provide support for that out of the box. While you could use something like autograd (see this example), TensorFlow has been built from the scratch with that in mind, and supports it very well. Similar frameworks include Theano or PyTorch. Using a framework like TensorFlow comes with its own additional perks too, like GPU support (again, not impossible with NumPy, but not straightforward), multiplatform support, distributed training... There is, of course, something of a trend factor with these things, and being backed by Google helps, but there are significant benefits to TensorFlow over plain NumPy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.