1

I am doing a school project and I wanted to apply genetic algorithms to neural networks. Specifically building a good neural network for the given problem. I noticed that there are some challenges involved with choosing the right parameters for a neural network so that it can train on some data quickly and efficiently and that it basically comes down to a guessing game. I wanted to build multiple neural networks in a genetic algorithm fashion to find the best initial starting setup to classify a given data set.

I am looking at using the Java library Neuroph, specifically extending the multilayer perceptron implementation. In the library there is the file MultilayerPerceptron.java under the path org.neuroph.nnet and it has the method createNetwork. Here is the implementation of that method

private void createNetwork(List<Integer> neuronsInLayers, NeuronProperties neuronProperties) {

    // set network type
    this.setNetworkType(NeuralNetworkType.MULTI_LAYER_PERCEPTRON);

    // create input layer
    NeuronProperties inputNeuronProperties = new NeuronProperties(InputNeuron.class, Linear.class);
    Layer layer = LayerFactory.createLayer(neuronsInLayers.get(0), inputNeuronProperties);

    boolean useBias = true; // use bias neurons by default
    if (neuronProperties.hasProperty("useBias")) {
        useBias = (Boolean)neuronProperties.getProperty("useBias");
    }

    if (useBias) {
        layer.addNeuron(new BiasNeuron());
    }

    this.addLayer(layer);

    // create layers
    Layer prevLayer = layer;

    //for(Integer neuronsNum : neuronsInLayers)
    for(int layerIdx = 1; layerIdx < neuronsInLayers.size(); layerIdx++){
        Integer neuronsNum = neuronsInLayers.get(layerIdx);
        // createLayer layer
        layer = LayerFactory.createLayer(neuronsNum, neuronProperties);

        if ( useBias && (layerIdx< (neuronsInLayers.size()-1)) ) {
            layer.addNeuron(new BiasNeuron());
        }

        // add created layer to network
        this.addLayer(layer);
        // createLayer full connectivity between previous and this layer
        if (prevLayer != null) {
            ConnectionFactory.fullConnect(prevLayer, layer);
        }

        prevLayer = layer;
    }

    // set input and output cells for network
    NeuralNetworkFactory.setDefaultIO(this);

    // set learnng rule
    //this.setLearningRule(new BackPropagation(this));
    this.setLearningRule(new MomentumBackpropagation());
    // this.setLearningRule(new DynamicBackPropagation());

    this.randomizeWeights(new NguyenWidrowRandomizer(-0.7, 0.7));
}

So I see many factors that can be tweaked in the building of the network. I am am trying to determine what I might make the genotype of my algorithm.

So the factors that I see that I could tweak are first the number of hidden layers and the number of hidden layer neurons; the input and output layer size is determined by the problem. The neuron properties of the neurons in the network, (specifically, can different neurons have different activation functions?). Possibly the connections between neurons; do all neurons have to be connected to all others in the layers above and below, can some skip a layer? The learning rate of the learning rule used, possibly use a different learning rule for different neurons (pretty sure that's a stupid question).

So overall, what are the factors of a neural network that can be tweaked on the initial setup that would effect how fast and efficiently the network learns? I guess very specifically do all neurons in a neural network have to be identical in operation?

EDIT: Through some search-fu I have found I guess the name of this type of network a compositional pattern producing network CPPN. The main idea being called neuroevolution. An algorithm that implements these ideas is the neuroevolution of augmenting topiologies NEAT. I am looking more into these pages and the links to the papers to try and answer my main questions above. But the question is still mostly open.

1 Answer 1

0

There are a million different things you can tweak with neural networks. They are not a rigid algorithm. All of the things you mentioned are valid. You can tweak the layer numbers and sizes, you can mix and match activation functions, you can have skip layer connections, and tune the learning rate.

These things are called "hyper-parameters", to distinguish them from regular parameters, i.e. the weights and connections of the neural network. They are set by lots of experimentating to see what works best.

However it's very easy to accidentally overfit. To get an NN which works very well on the training or validation data, but doesn't do well at all on new data. Use something like cross-validation to split the data into different parts, and test on each different part.

The main parameters are the number and size of the layers, and regularization parameters (e.g. weight decay or jitter.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.