What Is Machine Learning?

Machine Learning (ML), also known as Autonomous Learning, is a branch of Artificial Intelligence (AI) that aims to build systems able to learn on their own from a data set, without being explicitly programmed. 

For these systems to learn on their own, a series of techniques and algorithms capable of creating predictive models, behavior patterns, etc. are used. Although the current literature has no concrete, bounded list of the techniques and algorithms that belong to ML (even if some techniques are clearly part of this field), we can say that ML covers any problem-solving process based, more or less explicitly, on a rigorous application of statistical decision theory. It is therefore quite common for ML to overlap with statistics. ML, unlike statistics, focuses on studying the computational complexity of problems, since many of them are NP-complete (or NP-hard), so the challenge of ML is to design feasible solutions for this kind of problem.

Let’s look at two definitions of ML:

How the System Learns: Regression and Classification

As mentioned in the definition of ML, this field must create systems that are able to learn on their own without being explicitly programmed, in order to predict future events, make recommendations, classify items, events, tags, etc. So, after a learning phase, we will have an “expert system” that, given a certain input, will give us an output (prediction, recommendation, classification, etc.) as the result of applying a regression or classification function (which the system must learn) to the input data.

Two examples of expert systems created after applying some ML technique(s) are the following: first, an expert system for predicting football pools (classification) which, given the names of the home and away teams, returns one of the three options of the pool (1, X, 2); and second, an expert system for calculating calories burned (regression) while running, which, given the person’s weight, the running time and the speed as input, returns the number of calories burned (0 <= calories < ∞).

As we have seen, these systems have two ways of providing a result: first, classification, which returns a finite set of possible results, generally small (y={0,1}, y={1,X,2}, y={yes,no}); and second, regression, which returns an arbitrary value (a real number, a vector of real numbers, strings of symbols, etc.).

Now that we have explained what regression and classification are, let’s look at a more formal definition of each:

Now that we know the kind of expert systems we want to get after applying some ML technique(s), we need to see how these systems learn to obtain that classification or regression function, which from now on we will call the “hypothesis”.

For systems to learn, we need a learning or training data set (input and output data) that is used to obtain a hypothesis (a model or function) that generalizes that data properly. When we talk about “generalizing”, we mean predicting the output from new input data (test data) that is different from the training data.

We can see this learning process for obtaining a hypothesis outlined in the following image. First, we have a data set that we will use to train or teach the system. By applying one of the ML techniques, we will obtain a hypothesis that, a priori, should be the best function that fits and generalizes the training data. To understand this diagram, let’s suppose we are in the case of supervised learning (which we explain in the Types of Learning section), where for each input of the training data we know what its output will be. So, to obtain the hypothesis in this kind of learning, we need to find a function that fits that training data; in other words, one that minimizes the “empirical error”, which is the error measured after applying the hypothesis to the training data. What we really care about is that the hypothesis has the smallest possible error on the test data, but we assume that the training data is a sample representative enough that the error made on the test data will be similar to the empirical error.

In short, the goal of these techniques is to find the function or model (hypothesis), among the whole set of functions or models, that best fits and generalizes the training data, so we can apply it to the test data and get a prediction, recommendation, classification, etc.

Overfitting and Underfitting

One of the most important points in ML is achieving correct learning from the training data, obtaining a hypothesis able to generalize properly to new input data. On the other hand, obtaining a hypothesis as the result of overfitting or underfitting (overgeneralization) the training data will (most likely) make the output for new input data have a very high error, which means the prediction will not be correct.

To see the meaning of overfitting and underfitting in more detail, let’s take the case of our expert system for predicting football pools: We know that FC Barcelona is one of the most powerful teams in the world and, obviously, in the Spanish league. On the other hand, there are more modest teams, such as Rayo Vallecano, which concedes a lot of goals and loses more matches than it wins; but by chance, in the training data we have three results in which Rayo Vallecano beat FC Barcelona at their stadium. Also, in most of the training data, the football matches played by FC Barcelona end in a win; and another relevant fact is that 60% of the matches in the training data end in a home team win (result = 1). Given this data, what would the prediction be for the match “FC Barcelona-Rayo Vallecano”? 

This is only a summary of the data in the training set, but a person (whether or not they are a football expert), looking at the training data, could predict that the result of this match would be a “1” (FC Barcelona wins), since FC Barcelona is one of the most powerful teams, wins most of its matches, faces a team that loses more matches than it wins, and also plays at home (60% of matches are won by the home teams). Although ML learning algorithms cannot reason like this (since they work mathematically), we do expect the result to be coherent and similar to what a human would predict. But what result would the system give if overfitting or underfitting has occurred?

1. Overfitting ("FC Barcelona-Rayo Vallecano = Rayo Vallecano wins”): Since, by chance, the training data set only has 3 matches between these teams and they are all wins for Rayo Vallecano, by fitting myself to the training data I say that Rayo Vallecano wins, because that is the information available.
2. Underfitting ("FC Barcelona-Rayo Vallecano = FC Barcelona wins”): Because of the overgeneralization, the hypothesis outputs that the home team will win, since 60% of the results are home wins; so in this case the result would be correct.
3. Underfitting ("Rayo Vallecano-FC Barcelona = Rayo Vallecano wins”): By analogy with the previous case, since Rayo Vallecano is the home team, Rayo Vallecano would win.

The point of this long example is that the hypothesis obtained does not need to predict 100% of cases correctly, but we do have to demand, to some extent, that it generalizes properly, even if that means making occasional mistakes. In other words, we have to be able to find a hypothesis with a balance between overfitting and overgeneralization.

Let’s now look graphically at the concept of overfitting and underfitting for classification and regression problems:

Obtaining a classifier that approximates f:R²→{*,x}:

Overfitting	OK	Underfitting

Regression model that approximates f:R→R:

Overfitting	OK	Underfitting

Let’s see, for example in the case of regression, the error we would make if a new input data point (the red dot) arrived, for each of the functions obtained:

Overfitting	OK	Underfitting

You can see that the error made on a test data point is larger when overfitting or underfitting occurs, and smaller when the data has been generalized correctly, although obviously there is still a small error.

Types of Learning

Depending on the data we have to train the system, we can apply one type of learning or another. The most common types of learning are listed and explained below:

1. Supervised learning: A type of learning in which we have complete information about the training data; that is, both the input data and its output. It is the type of learning that gives the best results, since it has the most information.
2.  Unsupervised learning: A type of learning in which only the input data is available, and the goal is to obtain information about the structure of the output domain.
3. Semi-supervised learning: A hybrid type of learning between supervised and unsupervised learning.
4. Adaptive learning: A type of learning that starts from a previous model whose parameters are modified or adapted using the new training data.
5. Online learning: In this type of learning there is no clear distinction between the test phase and the training phase. The system learns (usually from scratch) through the prediction process itself, with human supervision that consists of validating or correcting each output based on the input.
6. Reinforcement learning: A hybrid type of learning between online learning and semi-supervised learning in which the supervision is incomplete; usually information of the type {yes,no}, {0,1}, {reward,punishment}. It is a type of learning based on the “argumentum ad baculum”, commonly used in animal training.
    

Methods for Evaluating a System

To evaluate the hypotheses obtained after applying some ML technique, we need a data set (labeled or not) to generate the best possible hypothesis and minimize the empirical error. Given a data set, we can list the following evaluation methods, depending on how the training and test data are split:

1. Resubstitution: A very optimistic method in which all the available data is used as both test and training data.
2. Hold Out: This method splits the data into two subsets: one for training and one for testing. The problem with this method is that the test data is wasted when obtaining the hypothesis.
3. Cross Validation: This method randomly splits the data into ‘N’ blocks. Each block is used as the test set for a system trained on the remaining blocks. The drawback of this method is that it reduces the amount of training data when each block is large.
4. Leave One Out: This method uses each individual data point as the only test data for a system trained on all the data except that test point. It is similar to cross validation, but in this case the computational cost is very high because of the number of learning phases that must be run.
    

Machine Learning Techniques

It is really hard to build a hierarchy or classification of ML techniques based on the problems they can solve. For example, some very specific techniques, like K-means or Expectation-Maximization (EM), are clearly used for clustering problems, while techniques like Support Vector Machine (SVM) or even Neural Networks can fit both regression and classification problems, although they are more typical of classification than regression. Regardless of how each technique can be classified, what is obvious is that ML professionals should make an effort to learn as many techniques as possible, and learn them well, so they have enough training to tackle and solve problems using the technique(s) that best fit.

Below we show a classification of ML techniques and problems, partly based on the classification proposed by the English Wikipedia.