6

I tried to build my own custom layer in tensorflow/keras that enforces the layer to be symmetric and what I ended up with is the following:

import tensorflow as tf
from tensorflow.python.framework.ops import enable_eager_execution
enable_eager_execution()

class MyDenseLayer(tf.keras.layers.Layer):
    def __init__(self, num_outputs):
        super(MyDenseLayer, self).__init__()
        self.num_outputs = num_outputs

    def build(self, input_shape):
        X = tf.random.uniform([int(input_shape[-1]),self.num_outputs],minval=0,maxval=1,dtype=tf.dtypes.float32,)
        k = tf.Variable(X, name="kernel")
        self.kernel = 0.5 * (k+tf.transpose(k))

    def call(self, input):
        return tf.matmul(input, self.kernel)

layer = MyDenseLayer(5)
print(layer(tf.ones([3, 5])))
print(layer.trainable_variables)

So far, so good. What I don't understand this: why does the last line

print(layer.trainable_variables)

give me an empty list:

[]

I thought that layer.trainable_variables would show me what my matrix looks like so that I could check whether it is symmetric or not.

1 Answer 1

3

You need to add variables using add_weight and then call build() method to create this variable. Alternatively, instead of calling build() directly you can pass an input (as you do in your question) and it will call implicitly the build() method.

import tensorflow as tf
from tensorflow.python.framework.ops import enable_eager_execution
enable_eager_execution()

class MyDenseLayer(tf.keras.layers.Layer):
    def __init__(self, num_outputs):
        super(MyDenseLayer, self).__init__()
        self.num_outputs = num_outputs
    def build(self, input_shape):
        def initializer(*args, **kwargs):
            X = tf.random.uniform([int(input_shape[-1]),self.num_outputs],minval=0,maxval=1,dtype=tf.dtypes.float32,)
            kernel = 0.5 * (X+tf.transpose(X))
            return kernel
        self.kernel = self.add_weight(name='kernel',
                                      shape=(input_shape[-1], self.num_outputs),
                                      initializer=initializer,
                                      trainable=True)
        super(MyDenseLayer, self).build(input_shape)

    def call(self, input_):
        return tf.matmul(input_, self.kernel)

layer = MyDenseLayer(5)
layer.build((5, )) # <-- example of input shape
print(layer.trainable_variables)
# [<tf.Variable 'kernel:0' shape=(5, 5) dtype=float32, numpy=
# array([[0.04476559, 0.8396935 , 0.42732996, 0.75126845, 0.7109113 ],
#        [0.8396935 , 0.46617424, 0.71654373, 0.5770991 , 0.38461512],
#        [0.42732996, 0.71654373, 0.75249636, 0.28733748, 0.6064501 ],
#        [0.75126845, 0.5770991 , 0.28733748, 0.9417101 , 0.61572695],
#        [0.7109113 , 0.38461512, 0.6064501 , 0.61572695, 0.6960379 ]],
#       dtype=float32)>]
9
  • Looks good. Does this layer stay symmetric after training? It seems like it is symmetric from the beginning but will change its property through backpropagation.
    – xabdax
    Commented Apr 27, 2019 at 21:32
  • 1
    Not the layer, but the weights of the layer! The layer is referred to the what your layer outputs (the return of the call method). The weights of the layer will keep the same size as you defined it in build method. It won’t change its size. The output of the layer may depend on the batch size though (the first dimension).
    – Vlad
    Commented Apr 27, 2019 at 21:38
  • Ok, I see. I tried to use the modified build and implement it in one of the existing keras layers (github.com/keras-team/keras/blob/…) but I am having issues with the output shape as in MyDenseLayer and the given parameters of the class Dense. Do you have any suggestions about how to implement your initializer in the class Dense? Do you think that is even a good idea?
    – xabdax
    Commented Apr 27, 2019 at 21:51
  • To implement the initializer just replace num_outputs with units. But I think it is wrong idea since the distribution of your initializer hasn’t changed. It is still uniform[0, 1] so just use built-in uniform initializer.
    – Vlad
    Commented Apr 27, 2019 at 22:13
  • So back to a previous question because I am not sure if I understood correctly: do you think that the weights will stay symmetric after backpropagation? The dimensions will definitely stay the same.
    – xabdax
    Commented Apr 27, 2019 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.