-1

I have two networks. D->C1,R1->P1->C2,R2->loss and D->C1,R1,C2,R2->P1->C3,R3->loss.

D is data, C for Conv, R for Relu and P for Pool.

I trained first network from scratch and since two networks have only C2,R2 layers are different, I used all weights from first network and train only C2. C1 and C3 are set with lr_mult=0.So their weights are not updated.

My query is how can I know C2 has optimum weight to stop training.

Say network 1 has mAP=0.85 after 5 epochs. When I train network 2, it has mAP=0.86 just after 1 epoch.

When should I stop training in such case?

1 Answer 1

1

Unfortunately, neural network optimization in general is non-convex, so it is impossible to know if a particular local minima is a global minima [*] The fact that you have pretrained weights isn't particularly relevant [**] . The type of solver has an effect

That said, there are some criteria that are occasionally used heuristically. Importantly: use a testing set, not a training set for evaluation (and then use a separate validation set when checking your performance).

  1. Change in MSE on the testing set plateus
  2. Cross valdiation [***]
  3. The learning rate vanishes (depends on your solver)
  4. A fixed number of iterations

Here's one slightly older survey, though results tend to always be empirical


Additionally, Goodfellow makes the following important recommendation about making sure your parameters are as optimal as possible regardless of your criteria:

Every time the error on the validation set improves, we store a copy of the model parameters. When the training algorithm terminates, we return these parameters, rather than the latest parameters. The algorithm terminates when no parameters have improved over the best recorded validation error for some pre-specified number of iterations



Footnotes

[*] There are other conditions (e.g) that might provide this information, but none of them apply

[**] I'm not aware that there's research on this one way or another, but I would suspect that it would actually make the problem harder, as you're starting from a pretty good local minima that may be hard to climb out of

[***] This is not the same as using cross validation to measure accuracy on a testing set or for model selection, see here

Not the answer you're looking for? Browse other questions tagged or ask your own question.