1

Actually these are 3 questions:

Which optimization algorithm should I use to optimize the weights of a multilayer perceptron, if I knew...

1) only the value of the error function? (blackbox)

2) the gradient? (first derivative)

3) the gradient and the hessian? (second derivative)

I heard CMA-ES should work very well for 1) and BFGS for 2) but I would like to know if there are any alternatives and I don't know wich algorithm to take for 3).

2
  • Do you have a particular problem that your trying to solve? I confess I can't think of any scenarios where you'd only use the error function, since most ANNs use functions that have easily computed derivatives. What's preventing you from using back-propagation?
    – zergylord
    Aug 29, 2011 at 20:35
  • I have two different categories of problems to solve: 1) a supervised learning task (brain computer interface data) 2) some reinforcement learning tasks with... a) contrinouos state and discrete action space. Here I have an error function and can apply backpropagation. b) continouos state and action space. I think I will have no direct error in this case because the input of the ANN is the state and the output is the action and I don't really know which action is optimal. But I have something like a fitness function (return).
    – alfa
    Aug 29, 2011 at 22:32

2 Answers 2

2

Ok, so this doesn't really answer the question you initially asked, but it does provide a solution to the problem you mentioned in the comments.

Problems like dealing with a continuous action space are normally not dealt with via changing the error measure, but rather by changing the architecture of the overall network. This allows you to keep using the same highly informative error information while still solving the problem you want to solve.

Some possible architectural changes that could accomplish this are discussed in the solutions to this question. In my opinion, I'd suggest using a modified Q-learning technique where the state and action spaces are both represented by self organizing maps, which is discussed in a paper mentioned in the above link.

I hope this helps.

2
  • Thanks for your answer, but I don't want to change the topology. Actually, I have a small modification for an ANN and I want to test it with RL problems. In case you have a continuous state space and a discrete action space you would create a ANN that has the input (s, a) and the output Q(s, a), so you can generate a policy (choose the action that maximizes Q(s, a)) by calculating Q(s,a) for all a. For continuous a this does't work, because you had to check an infinite number of actions. So I would approximate pi(s,a) directly with the return of an episode as the fitness.
    – alfa
    Aug 30, 2011 at 14:25
  • 1
    Too few characters left... :) I think this link is great: homepages.cwi.nl/~hasselt/papers/RL_in_Continuous_Spaces/…. Especially the part about "Policy Search with Evolutionary Strategies".
    – alfa
    Aug 30, 2011 at 14:31
0

I solved this problem finally: there are some efficient algorithms for optimizing neural networks in reinforcement learning (with fixed topology), e. g. CMA-ES (CMA-NeuroES) or CoSyNE.

The best optimization algorithm for supervised learning seems to be Levenberg-Marquardt (LMA). This is an algorithm that is specifically designed for least square problems. When there are many connections and weights, LMA does not work very well because the required space is huge. In this case I am using Conjugate Gradient (CG).

The hessian matrix does not accelerate optimization. Algorithms that approximate the 2nd derivative are faster and more efficient (BFGS, CG, LMA).

edit: For large scale learning problems often Stochastic Gradient Descent (SGD) outperforms all other algorithms.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.