Introduction to Neural Networks: The Perceptron (Video)

Neural networks (in artificial intelligence) are a learning and automatic processing paradigm inspired by the way the nervous system of animals works. They are a system of interconnected neurons that work together to produce an output.

As we know, artificial intelligence tries to imitate natural intelligence, so in this case neural networks try to imitate the brain’s "architecture" (neurons + connections) that gives people intelligence. It’s true that the term intelligence is pretty ambiguous, but here, when we talk about neural networks, we’ll consider intelligent those people who, knowing about a certain subject, are able to give an answer to a given problem (calculation, classification, etc.). For example, if we know a little about animals, we can classify an animal (given a photo) as a mammal, bird, reptile, etc. That is, somehow we have learned the difference between those animals and we know how to classify them. This is an example of what a well-trained (or well-taught) artificial neural network can do: given a photo of an animal, it is able to classify it. Of course, this neural network can fail the same way humans fail, but the idea is that a well-trained neural network is able to predict the same way people do.

Another question that may come up is how neural networks learn. Well, they do it the same way humans do. There are two types of learning: supervised and unsupervised learning. The names are pretty descriptive, so we know that supervised learning happens when we are taught by a teacher, master, etc., and unsupervised learning happens when we learn by ourselves.

In this post we’re going to explain the architecture of an artificial neuron and the way neurons connect to each other. We’ll also explain the perceptron and its concepts so that, as far as possible, we can choose the best architecture for our neural network. All of this falls under supervised learning, since the perceptron is a neural architecture that needs to be trained or taught.

Making a big abstraction of the human brain, we say the brain is made up of a set of neurons connected to each other. The human brain has around 10¹¹ neurons and about 10¹⁴ connections. Each neuron is connected to another 1K-200K neurons, so it’s no surprise that there are so many connections. As general knowledge, and to later see the structure of an artificial neuron, here is an image of a human brain neuron and of how neurons connect to each other:

              

These images don’t give us much information from an AI point of view, but it’s interesting to see what the connections mean and to know what each neuron does on its own. With this abstraction, we can consider that a neural network is made up of a number of simple elements, the neurons, and that each of these neurons performs a very simple operation (it computes a simple mathematical function based on the value of its inputs) whose result is passed on to the next neuron as its input (hence the connections). That said, here is an image of an artificial neuron:

As we can see, an artificial neuron has 'n' inputs (e1,…,en) that will have a certain numeric value. It also has what is called a "bias connection" (b0), which is a constant value for each neuron; and each input has a weight denoted as 'Wi'. With the values of the inputs and the weights, a value is computed that in the image is called input (although it’s usually called "net"), which is the sum of each input multiplied by its weight. Once we have that "Input" value, a certain output function is applied, known as the activation function, which is usually one of the ones shown below:

The identity function is also usually called the linear function, and for the step function the ranges can vary depending on how you define it. Once we’ve seen the activation functions and the architecture, it’s pretty intuitive that the output of an artificial neuron can be connected as the input of another neuron, and so we can build a neural network known as a perceptron, which will have at least one input neuron connected to an output neuron.

Next we’ll look at an example of a neural network that gives us the solution for the AND, OR and XOR logic gates, and see the architecture we need depending on our requirements.

AND gate

The AND gate returns a '0' if any of its inputs is '0' and returns a '1' if both inputs are '1'. What our neural network is going to do is split the solutions into two planes (see the next image). Whatever "falls" on one side of the plane will output a '0' and whatever falls on the other side will output a '1'.

As we can see in the image, whatever is above the green line will output a '1' and whatever is below the green line will output a '0'. Now, knowing that we have to draw a "discriminant" line to define two planes, how many neurons will our perceptron need to solve this problem? Well, two input neurons and one output neuron will be enough, since a single neuron is able to tell the two planes apart. In the following image we can see the weights the output neuron must have to solve the AND gate:

The input neurons haven’t been drawn in this image, since all they would do is output the same value as their input, with all their weights set to one and a bias connection equal to '0' (b0=0).

We can test it to see that the result is correct:

• e1=0; e2=1; input = (0·1)+(1·1)+(1·(-1.5)) = 1-1.5 = -0.5 -> F(-0.5) = 0; because -0.5<0;
• e1=1; e2=1; input = (1·1)+(1·1)+(1·(-1.5)) = 1+1-1.5 = 0.5 -> F(0.5) = 1; because 0.5>0;

OR gate

The OR gate is the same as the AND gate: we only need to "draw a line" to tell the outputs apart, so the architecture will be the same as the AND but with different weights:

                

• e1=0; e2=0; input = (0·0)+(0·0)+(1·(-0.5)) = -0.5 -> F(-0.5) = 0; because -0.5<0;
• e1=1; e2=0; input = (1·1)+(0·1)+(1·(-0.5)) = 1+0-0.5 = 0.5 -> F(0.5) = 1; because 0.5>0;

XOR gate

A somewhat more complex example is the XOR gate, since it needs two hyperplanes to tell the outputs apart:

This means that with a single neuron we won’t be able to find a solution to the problem. Since there are two hyperplanes, we’ll need a perceptron with two neurons in the hidden layer. So the architecture of this perceptron will have two neurons in the input layer, two neurons in the hidden layer and one neuron in the output layer. In the end, we only need to work out how many neurons we need in the hidden layer, since the number of neurons in the input and output layers is determined by the number of inputs and outputs of our problem. Below is the neural network that solves the XOR problem:

In this architecture, we can see that the input neurons output the same value as their input, since their activation function is a linear function. Next we trace how the outputs are obtained from the inputs:

---

e1=0; e2=0;

Neuron3 input = (e1·w11)+(e2·w21)+(b30·w30) = (0·1)+(0·1)+(1·(-0.5)) = -0.5 -> F3(-0.5) = 0;

Neuron4 input = (e1·w12)+(e2·w22)+(b40·w40) = (0·1)+(0·1)+(1·(-1.5)) = -1.5 -> F4(-1.5) = 0;

Output-> Neuron5 input = (F3·w31)+(F4·w41)+(b05·W05) = (0·1)+(0·(-1))+(1·(-0.5)) = -0.5 -> F5(-0.5)=0

---

e1=0; e2=1;

Neuron3 input = (e1·w11)+(e2·w21)+(b30·w30) = (0·1)+(1·1)+(1·(-0.5)) = 0.5 -> F3(0.5) = 1;

Neuron4 input = (e1·w12)+(e2·w22)+(b40·w40) = (0·1)+(1·1)+(1·(-1.5)) = -0.5 -> F4(0.5) = 0;

Output-> Neuron5 input = (F3·w31)+(F4·w41)+(b05·W05) = (1·1)+(0·(-1))+(1·(-0.5)) = 0.5 -> F5(0.5)=1

---

e1=1; e2=0;

Neuron3 input = (e1·w11)+(e2·w21)+(b30·w30) = (1·1)+(0·1)+(1·(-0.5)) = 0.5 -> F3(0.5) = 1;

Neuron4 input = (e1·w12)+(e2·w22)+(b40·w40) = (1·1)+(0·1)+(1·(-1.5)) = -0.5 -> F4(-0.5) = 0;

Output-> Neuron5 input = (F3·w31)+(F4·w41)+(b05·W05) = (1·1)+(0·(-1))+(1·(-0.5)) = 0.5 -> F5(0.5)=1

---

e1=1; e2=1;

Neuron3 input = (e1·w11)+(e2·w21)+(b30·w30) = (1·1)+(1·1)+(1·(-0.5)) = 1.5 -> F3(1.5) = 1;

Neuron4 input = (e1·w12)+(e2·w22)+(b40·w40) = (1·1)+(1·1)+(1·(-1.5)) = 0.5 -> F4(0.5) = 1;

Output-> Neuron5 input = (F3·w31)+(F4·w41)+(b05·W05) = (1·1)+(1·(-1))+(1·(-0.5)) = -0.5 -> F5(-0.5)=0

This is how we see how to solve a problem using a perceptron with a hidden layer, and how problems like this are solved with neural networks. This last example is very simple but very instructive, and having seen it we can say that to learn the XOR gate we only need 5 of our 10¹¹ brain neurons 😉 so we have plenty left over for lots of other things.

Next we’ll show a tool called Neuroph Studio that lets you work with neural networks very easily. We’ll show this same XOR example (using a different activation function) so you can see how neural networks are trained. This tool can do much more complex things than a neural network that solves the XOR problem, but with this example you can get comfortable for future projects:

Neural network architecture with its weights:

Conclusions:

Neural networks are a really cool area of artificial intelligence, since they aim to nothing less than imitate the behavior of the human brain. In the end, what we do when we create a neural network and train it is build a model that gives us an answer based on the input we give it. If we use a neural network to create a model that will never change (for example, an animal classifier), neural networks are a good solution, but if on the contrary we create a model that changes over time (for example, one that predicts the behavior of the stock market or the result of a soccer match), neural networks are a very bad choice, because we’d have to keep updating the models (training them) and most likely we’d have to modify the architecture of the neural network and train it again.

Even though this post has shown how neural networks work and explained their concepts, it has to be said that there are better techniques (like those of Machine Learning) that do the same thing as neural networks, so it’s good to know how neural networks work to solidify concepts, but the perceptron is already a bit outdated.

The same can’t be said for Kohonen networks (or self-organizing maps), which are a very good solution when you need to apply unsupervised learning, although these networks have a different architecture from the one explained in this post.

In short, learn how neural networks work (and spend a few neurons on it, we have plenty 😉 ), but I recommend learning Machine Learning techniques to solve this kind of problem, since they are better than these.

Bibliography

Part of the information shown in this post was obtained from the classes of the course "Neural Networks and Applications" taught in the Master’s in "Computer Science and Technology" at the School of Computer Science of the UPM by professors Antonio Hernando and Soledad Delgado.