Deep Diaries · · 5 min read
Genetic Algorithms for Hyper-parameters Tuning
Genetic algorithms search for the best offspring (children) by mixing between the best parents, the best parents and children were selected by evaluation using a fitness function that we wish t
Originally published on Deep Diaries on Substack. Reproduced here as written.
Introduction
Today, neural networks (NN) invade our world from industry and medical fields to entertainment, As you can see there are too many NN architectures out there working well and achieving great accuracy day after another, but what if you want to create your own architecture? your own model to fit your problem or let’s think of a bad situation when all these architectures failed to solve your problem ?! Are you ready to build your own network!
Building a neural network is not an easy job to do as building a massive NN or a tiny one can result in a failure solving your problem.
So how to be sure that your architecture is the best ?!
For example, should you increase the number of hidden layers ?! Should you add some more convolution layers? Is increasing the number of neurons going to add value?
How do choose these values? Here hyper-parameters tuning comes into play.
Hyper-parameters tuning
Firstly, what is a hyper-parameter?
“In machine learning, a hyper-parameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are derived via training.” [1]
And what is hyper-parameter tuning?
“Hyper-parameter optimization or tuning is the problem of choosing a set of optimal hyper-parameters for a learning algorithm. A hyper-parameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned.” [2]
The disadvantage of hyper-parameter tuning
This process can be time-consuming as your searching space gets bigger.
To overcome this problem there are many ways to help you tune your hyper-parameters without testing every single combination in your searching space. One of these hyper-parameter tuning techniques is genetic algorithms (GA).
What are Genetic algorithms?
Genetic algorithms are inspired by natural evolution theory.
Simply genetic algorithms search for the best offspring (children) by mixing between the best parents, the best parents and children were selected by evaluation using a fitness function that we wish to maximize (ex: accuracy) or minimize (ex: loss).
Genetic algorithms structure
GA are working using what's called Genotype, it is a sequence of values where every value called a Gen the combination of these Genes together form one solution which called (Individual) which is one combination of different values of hyper-parameters, this is to facilitate applying the three main process of GA process which is:
Selection
Crossover
Mutation
Selection
Its name tells that we are going to select some values from a set of values. But how to select? according to which role? this is what we are going to discuss but first let me introduce the “population term” in the GA population means the set of all individuals before the selection process.
The aim of the selection process is to select the best N parents where N is an integer you choose.
The selection process follows one of the following methods:
Roulette wheel Selection (fitness proportionate selection):
To apply this method we should calculate the fitness value for each individual in the population and the probability of each individual which is equal to the fitness value divided by the sum of the fitness values for all individuals in the population, each individual was assigned to a space on the roulette wheel equals to its probability and all we need to do is spinning the wheel to get our winner, you can think of assigning each individual space on the wheel as repeating each individual according to its probability and pick the winner randomly.
Stochastic universal sampling
Is a modified version of the roulette wheel with the same technique but instead of picking only one winner we can pick N number of winners at once.
Rank-based selection
This method is also a modified version of the roulette wheel but the only difference the is doesn’t use the fitness value to calculate the probability, it uses the fitness rate to set a rank for each individual and use this rank to calculate the probability, this can be useful when a few individuals have much larger fitness values from the others.
Tournament selection
This method works by dividing the population into small groups and selecting one parent per group as the best of the group and repeating the process til find the best two parents
Crossover
After choosing the parents, we need to merge the chromosomes (crossover) to generate new offspring with the properties of each parent. Simply crossover is the process of splitting a parent's chromosomes into parts and interchanging these parts to form offspring.
1 - One point crossover: by splitting the parent’s chromosomes at some random points and interchange parts among the parents.
For example:
If parent one (P1) is 0011 and parent two (P2) is 1100 and the splitting point is 2 which means after the second element of the parent
2 - Two-point crossover: the same as one point crossover but using two splitting points
3 - uniform crossover: by choosing random points to be exchanged between parents this process was done by initializing a binary vector (alpha) where each 0 mean that the value in this location will be exchanged and 1 means the opposite,
Offspring 1 = (alpha * each value of parent 1) + ((1 - alpha) * each value of parent 2)
Offspring 2 = (alpha * each value of parent 2) + ((1 - alpha) * each value of parent 1)
Mutation
After initializing the population and selecting the parents and generating the offspring (new generation) there is no guarantee that these offsprings will do better than their parents, there is no genetic mutation to make a big difference, mutations happened in nature because of errors in copying the genes, a mutation in genetic algorithms could cause a leap in performance of your model by exploring the new small area in searching space.
Mutation can be done in two ways:
Randomly generate a new bit which causes changing the original value in 50% of the cases
Flipping the bit which changes the value in 100% of the cases
Typically mutation is applied by randomly setting a mutation probability for each value in the offspring and mutating those with a probability of less than 1%.
After Mutation, now we have our generation to evaluate so the next step is evaluating the new generation and then append the new generation to the old one (parents) and select the best parent for the next generation and do it over and over again till we reach our goal.
Summary
Genetic algorithms are inspired by evolution, trying to find the optimal solution for a problem in search space by selecting the best individuals of a population and applying crossover and mutation on these individuals to create a new generation of solutions. All of this with respect to some fitness functions that we are trying to minimize or maximize.
References