15

I have written the following convolutional neural network (CNN) class in Tensorflow [I have tried to omit some lines of code for clarity.]

class CNN:
def __init__(self,
                num_filters=16,        # initial number of convolution filters
             num_layers=5,           # number of convolution layers
             num_input=2,           # number of channels in input
             num_output=5,          # number of channels in output
             learning_rate=1e-4,    # learning rate for the optimizer
             display_step = 5000,   # displays training results every display_step epochs
             num_epoch = 10000,     # number of epochs for training
             batch_size= 64,        # batch size for mini-batch processing
             restore_file=None,      # restore file (default: None)

            ):

                # define placeholders
                self.image = tf.placeholder(tf.float32, shape = (None, None, None,self.num_input))  
                self.groundtruth = tf.placeholder(tf.float32, shape = (None, None, None,self.num_output)) 

                # builds CNN and compute prediction
                self.pred = self._build()

                # I have already created a tensorflow session and saver objects
                self.sess = tf.Session()
                self.saver = tf.train.Saver()

                # also, I have defined the loss function and optimizer as
                self.loss = self._loss_function()
                self.optimizer = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)

                if restore_file is not None:
                    print("model exists...loading from the model")
                    self.saver.restore(self.sess,restore_file)
                else:
                    print("model does not exist...initializing")
                    self.sess.run(tf.initialize_all_variables())

def _build(self):
    #builds CNN

def _loss_function(self):
    # computes loss


# 
def train(self, train_x, train_y, val_x, val_y):
    # uses mini batch to minimize the loss
    self.sess.run(self.optimizer, feed_dict = {self.image:sample, self.groundtruth:gt})


    # I save the session after n=10 epochs as:
    if epoch%n==0:
        self.saver.save(sess,'snapshot',global_step = epoch)

# finally my predict function is
def predict(self, X):
    return self.sess.run(self.pred, feed_dict={self.image:X})

I have trained two CNNs for two separate tasks independently. Each took around 1 day. Say, model1 and model2 are saved as 'snapshot-model1-10000' and 'snapshot-model2-10000' (with their corresponding meta files) respectively. I can test each model and compute its performance separately.

Now, I want to load these two models in a single script. I would naturally try to do as below:

cnn1 = CNN(..., restore_file='snapshot-model1-10000',..........) 
cnn2 = CNN(..., restore_file='snapshot-model2-10000',..........)

I encounter the error [The error message is long. I just copied/pasted a snippet of it.]

NotFoundError: Tensor name "Variable_26/Adam_1" not found in checkpoint files /home/amitkrkc/codes/A549_models/snapshot-hela-95000
     [[Node: save_1/restore_slice_85 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/restore_slice_85/tensor_name, save_1/restore_slice_85/shape_and_slice)]]

Is there a way to load from these two files two separate CNNs? Any suggestion/comment/feedback is welcome.

Thank you,

5 Answers 5

26

Yes there is. Use separate graphs.

g1 = tf.Graph()
g2 = tf.Graph()

with g1.as_default():
    cnn1 = CNN(..., restore_file='snapshot-model1-10000',..........) 
with g2.as_default():
    cnn2 = CNN(..., restore_file='snapshot-model2-10000',..........)

EDIT:

If you want them into same graph. You'll have to rename some variables. One idea is have each CNN in separate scope and let saver handle variables in that scope e.g.:

saver = tf.train.Saver(tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES), scope='model1')

and in cnn wrap all your construction in scope:

with tf.variable_scope('model1'):
    ...

EDIT2:

Other idea is renaming variables which saver manages (since I assume you want to use your saved checkpoints without retraining everything. Saving allows different variable names in graph and in checkpoint, have a look at documentation for initialization.

1
  • Thank you very much. Your first suggestion works fine for my case.
    – Amit
    Commented Feb 2, 2017 at 1:15
10

This should be a comment to the most up-voted answer. But I do not have enough reputation to do that.

Anyway. If you(anyone searched and got to this point) still having trouble with the solution provided by lpp AND you are using Keras, check following quote from github.

This is because the keras share a global session if no default tf session provided

When the model1 created, it is on graph1 When the model1 loads weight, the weight is on a keras global session which is associated with graph1

When the model2 created, it is on graph2 When the model2 loads weight, the global session does not know the graph2

A solution below may help,

graph1 = Graph()
with graph1.as_default():
    session1 = Session()
    with session1.as_default():
        with open('model1_arch.json') as arch_file:
            model1 = model_from_json(arch_file.read())
        model1.load_weights('model1_weights.h5')
        # K.get_session() is session1

# do the same for graph2, session2, model2
1

You need to create 2 sessions and restore the 2 models separately. In order for this to work you need to do the following:

1a. When you're saving the models you need to add scopes to the variable names. That way you will know which variables belong to which model:

# The first model
tf.Variable(tf.zeros([self.batch_size]), name="model_1/Weights")
...

# The second model 
tf.Variable(tf.zeros([self.batch_size]), name="model_2/Weights")
...

1b. Alternatively, if you already saved the models you can rename the variables by adding scope with this script.

2.. When you restore the different models you need to filter by variable name like this:

# The first model
sess_1 = tf.Session()
sess_1.run(tf.initialize_all_variables())
saver_1 = tf.train.Saver([v for v in tf.all_variables() if 'model_1' in v.name])
saver_1.restore(sess_1, weights_1_file)
sess_1.run(pred, feed_dict={image: X})

# The second model
sess_2 = tf.Session()
sess_2.run(tf.initialize_all_variables())
saver_2 = tf.train.Saver([v for v in tf.all_variables() if 'model_2' in v.name])
saver_2.restore(sess_2, weights_2_file)
sess_2.run(pred, feed_dict={image: X})
-1

I encountered the same problem and could not solve the problem (without retraining) with any solution i found on the internet. So what I did is load each model in two separate threads which communicate with the main thread. It is simple enough to write the code, you just have to be careful when you synchronize the threads. In my case each thread received the input for its problem and returned to the main thread the output. It works without any observable overhead.

1
  • could you provide an example of your solution?
    – Gabriele
    Commented Oct 1, 2020 at 16:04
-1

One way is to clear your session if you want to train or load multiple models in succession. You can easily do this using

from keras import backend as K 

# load and use model 1

K.clear_session()

# load and use  model 2

K.clear_session()`

K.clear_session() destroys the current TF graph and creates a new one. Useful to avoid clutter from old models / layers.

1
  • 2
    That works when you're trying to load the models one after another, but not when you want to load them together. Plus, that's Keras, the Tensorflow alternative is tf.reset_default_graph().
    – tsveti_iko
    Commented Jan 8, 2020 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.