Artificial Neural Networks are basically an attempt to make a computer model of the brain. The main objective is to develop a system to perform various computational tasks faster than the traditional systems with a capability of making decisions based on previous experiences. The Artificial Neural Networks (ANN) adopts the workings of the human brain in processing a combination of input stimuli to give an output. ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain.
Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it. The “signal” at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold
The neuron which is the basic block in an ANN is an imitation of how it looks like in the human brain as in the figure below
Its eyeless head is the neuron. It is connected to other neurons by those tentacles around it called dendrites and by the tails, which are called axons. Through these flow the electrical signals that form our perception of the world around us.
A gap exists between dendrites and axons so that when the signals travel through them, they do not touch. Neurons communicate with one another at junctions called synapses. At a synapse, one neuron sends a message to a target neuron—another cell. Most synapses are chemical; these synapses communicate using chemical messengers. Other synapses are electrical; in these synapses, ions flow directly between cells.
A Simplistic Model Of Artificial Neural Network
The figure below gives a highly simplistic overview of an Artificial Neural Network (ANN).
The inputs on the left side represent the incoming signals to the main neuron in the middle. In a human neuron, these could include taste or touch.
In your Neural Network, these inputs are independent variables. They travel down the synapses, go through the big yellow circle, then emerge on the other side as output values. The level of control that you could exert over the independent variables (inputs) is the main difference between the biological process and the ANN counterpart.
Now without trying to explain more of the concepts, let’s jump straight to a real life example and start learning ANN. Let’s predict the price of a property based on some of its properties as below. In the Fig. 2 above, let’s assume that the four inputs are related to the property and are
- Number of Bedrooms
- Area of the property (sq. meters)
- Age
- Distance from the city (km.)
These four properties will be our input layer and in the most simple form there are no hidden layers(neuron) and we have an output layer that simply predicts the price based upon the weights attached to the inputs.
These weights (or synapses in a biological counterpart) w1, w2, w3, etc could be anything like activation functions but are attached to each property, with property value multiplied by the weights and the summation of them all gives a rough prediction of the price. Of course, there’s probably many more parameters that define the price of the property, but for simplicity’s sake, we are just going to look at these four. A neural network in its very basic form has only the inputs (independent variables) and the output variables. Now without the hidden layers, these input variables will be weighed up by the synapses, and the output layer (house price) would be calculated. You could use pretty much any function even what we’re using now. We could use any of the activation functions introduced earlier or use logistic regression or squared function. The point here is that we would still get a price. Moreover, most of the machine learning algorithms can be represented in this form. Though as it is the neural network can still predict the house price but it has another layer, the hidden layer that gives it lots of flexibility and power. This hidden layer is the reason that the increased accuracy in the output (house price) prediction comes from.
Introducing the hidden layers
Lets introduce the hidden layer which gives the neural networks such power :
Let’s understand how that hidden layer gives us the extra power . The assumption that we make at this stage is that the neural network model is already trained and now we are going to plug in a property. Let’s understand how the neural network will deal with the input variables to calculate the output layer.
Case 1: We have all four variables on the left and the top neuron. Now all of the variables from the input layer have synapses connection to the each one of the neurons. Also these synapses have weight. Let’s assume that some of these weights have a non-zero value and some have a zero value as all of the inputs would not be valid for all of the neurons.
Fig 5 shows that the area (m2) of the property and distance from the city, is important to the top neuron whereas age and number of bedrooms are not. The correlation of these two input properties could be that the size of the property will be bigger the farther one goes away from the city. This is due to the assumption that the real estate gets cheaper if we move farther from the city resulting in bigger size property for the same price.
Case 2 (Fig. 6): Here we shall consider the middle neuron and it’s link to the three input properties: Number of bedrooms, Area of the property (m2) and Age (how old the property is?) and it does not care about the distance from the city. So what could be the scenario when this combination does fire up and activate this middle neuron. It could be that due to some demographics change due to availability of jobs for the society’s young who have 2-3 children and they would like big houses with lots of bedrooms. They prefer newer houses but don’t care about the distance from the city.
We have assumed earlier that the dataset had been trained long back (may be a day earllier or more). Also let think that it was trained in a suburb where the young ones with at two or more children who like newer houses which are big in size with a number of bedrooms (say 4 or more) and for them distance is not a problem. It is established that older properties normally fill this criteria of big sized houses with a more bedrooms, so this neuron has to find newer properties fulfilling the other two criteriae, all depending on the thousands of examples of properties on which it may have earlier been trained on.
The above two cases show how the power of neural networks as the neurons can combine multiple of parameters to make a brand new parameter into a brand attribute that helps with the evauation.
Finally, let’s look at the very bottom neuron which could only be dependent upon one parameter, age (of the property). Now we have to think what case would satisfy that condition that age (new or old) plays the critical role. We could think the older the property, the cheaper it will be as it’s worn out with age and more maintenance would be required. On the contrary, a new building will be more expensive, simply as it is new. But perhaps if a property is over a certain age (say > 100 yrs), then it will be classified as a historic property. Because of the fact that there is always a section of customers who love to buy hostoric things, these properties could command a much higher price than those say < 100 years old. This means the price of property drops as it grows older till 100 years. But as soon as it jumps over 100 years in age, it’s value increases suddenly as this has now become a historic property where people used to live hundreds of years ago. It means it tells a story and got a history behind it.
In conclusion, the hidden layer neurons could have been trained in datasets where they may have got activated by any combination of input parameters to get the output. One of them may find the combination of all four inputs to fit a scenario and get activated contributing to the output.
The Activation Function
It may be defined as the extra force or effort applied over the input to obtain an exact output. In ANN, we can also apply activation functions over the input to get the exact output. This function facilitates whether a signal gets passed on or not.
If weighted input values are the tomato ketchup, yoghurt and ginger garlic paste in the pot, the neuron would be the cooking black pot (looks like a drum eh..). The activation function is the open flame beneath that congeals the concoction into something new; the output value which could be a yummy chicken dish.
Definition of activation function:- Activation function decides, whether a neuron should be activated or not by calculating weighted sum and further adding bias with it. The purpose of the activation function is to introduce non-linearity into the output of a neuron.
Why do we need Non-linear activation functions :-
A neural network without an activation function is essentially just a linear regression model. The activation function does the non-linear transformation to the input making it capable to learn and perform more complex tasks. It doesn’t matter how many hidden layer we attach in neural net, all layers will behave the same way because the composition of two linear functions is a linear function itself. A neuron can not learn with just a linear function attached to it. A non-linear activation function will let it learn as per the difference w.r.t error.
Hence we need an activation function.
VARIANTS OF ACTIVATION FUNCTION :-
1). Linear Function :-
- Equation : Linear function has the equation similar to as of a straight line i.e. y = ax
- No matter how many layers we have, if all are linear in nature, the final activation function of last layer is nothing but just a linear function of the input of first layer.
- Range : -inf to +inf
- Uses : Linear activation function is used at just one place i.e. output layer.
- Issues : If we will differentiate linear function to bring non-linearity, result will no more depend on input “x” and function will become constant, it won’t introduce any ground-breaking behavior to our algorithm.
For example : Calculation of price of a house is a regression problem. House price may have any big/small value, so we can apply linear activation at output layer. Even in this case neural net must have any non-linear function at hidden layers.
2). Step (Threshold Function)
The x-axis represents the weighted sum of inputs in a step (threshold function). On the y-axis are the values from 0 to 1. If the weighted sum is valued as less than 0, the threshold function (TF) will pass on the value 0. If the value is equal to or more than 0, the TF passes on 1. It is a yes or no, black or white, binary function.
2). Sigmoid Function :-
- It is a function which is plotted as ‘S’ shaped graph.
- Equation :
A = 1/(1 + e-x) - Nature : Non-linear. Notice that X values lies between -2 to 2, Y values are very steep. This means, small changes in x would also bring about large changes in the value of Y.
- Value Range : 0 to 1
- Uses : Usually used in output layer of a binary classification, where result is either 0 or 1, as value for sigmoid function lies between 0 and 1 only so, result can be predicted easily to be 1 if value is greater than 0.5 and 0 otherwise.
3). Tanh Function :-
The activation that works almost always better than sigmoid function is Tanh function also knows as Tangent Hyperbolic function. It’s actually mathematically shifted version of the sigmoid function. Both are similar and can be derived from each other.Equation :-
f(x) = tanh(x) = 2/(1 + e-2x) - 1 OR tanh(x) = 2 * sigmoid(2x) - 1
- Value Range :- -1 to +1
- Nature :- non-linear
- Uses :- Usually used in hidden layers of a neural network as it’s values lies between -1 to 1 hence the mean for the hidden layer comes out be 0 or very close to it, hence helps in centering the data by bringing mean close to 0. This makes learning for the next layer much easier
4). Rectifier :- It is the most widely used activation function. Chiefly implemented in hidden layers of Neural network.
Equation :- A(x) = max(0,x). It gives an output x if x is positive and 0 otherwise.
Value Range :- [0, inf)
Nature :- non-linear, which means we can easily backpropagate the errors and have multiple layers of neurons being activated by the ReLU function.
Uses :- Rectifier is less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations. At a time only a few neurons are activated making the network sparse making it efficient and easy for computation.
In simple words, Rectifier learns much faster than sigmoid and Tanh function.