AI models have made great strides in recent years, and AI often has a quasi-magical perception. In this image, AI models are black boxes that somehow continuously learn from data it’s been provided and, on this basis, somehow responds to queries with answers (Fig. 1).

Fig. 1: Native view of AI models

This image is not wrong and it’s largely sufficient at the level of prompt engineering. But it’s vague, of course, and often contains the word “somehow.” This article lifts the hood and takes a look at how machine learning works in detail. I use the terms ML and AI synonymously, with AI referring to “large” models in all their vagueness.

Let’s begin by defining ML to give the topic some structure and distinguish it from similar, related fields: “Machine learning is the training of a model using statistical methods to predict the values of dependent variables based on input variables.”

Let’s look at the individual parts of this definition. The starting point for a machine learning project is typically the dependent variables, i.e., the values that the model is supposed to deliver. These could be sales and profit forecasts for the next quarter, or even a poem in dialect on a given topic, or the next move in a chess game. It’s important to have a clear idea of what the model should ultimately deliver because this determines all other aspects.

The input variables (or independent variables) are the variables that the model is supposed to derive its values from. There are often many possibilities for this, and the selection is a creative, technical process. For example, should previous business figures be used in weekly, monthly, or quarterly granularity to predict sales? How far back should the figures go? Should employee data be used, and is this even legally permissible? What about figures from competitors, etc.? Experience is needed, and different variables are often tested and lessons learned along the way.

The third important aspect is machine learning’s predictive nature. It involves predicting the dependent variables for new, unknown contexts. For example: training an AI for facial recognition that evaluates a live video feed or predicts sales figures for future quarters. The limits on the quality of predictions based on new, unknown input data and their verification can arise in many ML areas.

Fourth, ML means tackling problems with statistical methods. This is by no means the only option, and sometimes it’s clearly a poor choice. When a bank calculates a new account balance based on deposits and withdrawals, this must be done using deterministic algorithms, not statistical approximation. For many other problems, it works well to find and implement heuristics through careful thinking – plenty of video game “AIs” work this way. ML is not about completely “solving” a problem domain, but rather about approximations. For some variables – lottery numbers, for example – statistical methods are fundamentally unsuitable. At the very beginning of an ML project, you should check if statistically validated approximate solutions are suitable for the domain and if they’re the approach of choice.

Once you’ve decided to solve a problem statistically, you must choose a model. This is a formula for calculating the dependent variables from the input variables, but it contains parameters that can be adjusted. The model can be very simple—in an oversimplified extreme case, a constant percentage growth in sales from quarter to quarter, with the percentage as the only parameter—or it can be as complicated as you like, e.g., a deep neural network with millions of parameters. The model choice is a human decision based on experience and domain requirements. Deep neural networks are just one of many options, it is not automatically the best.

Sixth, ML ultimately means training the model, i.e., optimizing the parameter values using statistical methods. This is the part where ML “learning” happens and is done by iteratively adjusting the parameters. In the simplest case, training can be done using existing training data (supervised learning). But there are also approaches for when there is not enough training data (unsupervised learning, reinforcement learning) – more on that later.

A simple example

Let’s consider a simple example. The mathematics and fundamentals are the same as for deep neural networks and other large models, but a small example is clearer.

Let’s assume there is a variable y that we want to predict as a function of a variable x (I find this easier than giving the variables pseudo-illustrative technical names). y is therefore the dependent variable and x the only independent variable. We have sample data for the relationship between the variables; Figure 2 shows 200 data points.

Fig. 2: An example data set as a basis for training

Let’s also assume that we have examined the domain in detail and decided that we want to predict the relationship between y and x statistically rather than algorithmically. This makes the problem a candidate for ML. The next step is to find a model whose parameters we can then adjust to the specific values. For this, we choose a third-degree polynomial: ŷ(x) = aꞏx³ + bꞏx² + cꞏx + d. Here, ŷ denotes the predicted value for the variable y. The function has four parameters, a, b, c, and d, as well as the independent variable x. Choosing this model means that we assume that the relationship between x and y can be described well or well enough by this polynomial (with a suitable choice of parameter values).

Choosing a third-degree polynomial is not mandatory. For example, based on our knowledge of the domain, we could have decided that b and d are always 0 and removed the corresponding terms from the model. Or we could have assumed a fifth-degree polynomial or a sine curve. Or a deep neural network.

The choice of model is often a trade-off between complexity, comprehensibility, and accuracy, and a good choice is based on an understanding of the domain or assumptions about it. In practice, you often start with one model, learn more about the domain, and start over with a newly trained model.

Training the model

Now we need to find “good” values for the parameters a, b, c, and d, i.e., values with which the model makes the “best possible” predictions for ŷ. This trains the model. Parameter values are changed iteratively until they fit. This must be strictly separated from model usage, in which the parameters are fixed and the model always processes new input values. We start with random values:

• a = 0,15434
• b = 0,75297
• c = -0,08099
• d = 0,57356

Figure 3 shows the predictions in light green and the data points in purple. The predictions have no relation to the data, which was to be expected with random parameter values. If the iterative improvement of the values works, then the starting values should not matter.

Fig. 3: The initial parameter values are not optimal yet

Before we iteratively “improve” the parameter values using any algorithm, we need to specify what “good” means. In our example, we want the prediction to be as accurate as possible on average—a plausible and very common goal. But we could also optimize it so that the maximum error is as small as possible, for example.

Mathematically, our choice means that the mean squared error (MSE) is as small as possible: for each data point, we take the difference between the actual and predicted values, square this difference (to become independent of the sign and because it dampens the influence of small differences), and calculate the mean value across all data points (Fig. 4).

Fig. 4: Definition of mean square error

The smaller the MSE, the better our model fits the actual data. A value of exactly 0 would mean a perfect match, which of course, does not occur in real problems. Negative values cannot occur because the MSE is the sum of squares.

Figure 5 shows the magnitude of the square error for each data point (as blue bars at the bottom of the diagram) for the random initial values of the model. There is a range in which the predictions are quite close to the real data, but especially for small values of x, they are far apart. The mean square error is MSE = 2.318 – so the mathematical description fits with the observation that the predictions are still terrible.

Fig. 5: The square error varies for different data points

With these preparations in place, we can now enter the training loop (Fig. 6):

• We apply the model with the current parameter values to our training data to calculate the current MSE (we have already done this).
• Then, based on this, we calculate changes for the parameters that reduce the MSE.
• Finally, we change the model parameters accordingly to start a new run with the new parameter values.

The second step, calculating the parameter changes that reduce the error, requires some mathematics. This is explained separately at the end of the article. The rest of the procedure still makes sense even if this step is considered a black box.

In any case, this black box delivers the following changes for the parameters in the first step:

• a: 0.15434 ↦ 0.17299
• b: 0.75297 ↦ 0.72049
• c: -0.08099 ↦ -0.0687
• d: 0.57356 ↦ 0.54846

We will start the second iteration (epoch) with these values. Figure 7 shows how the models—the colored curves—continue to converge on the training data. After 2,000 epochs, the model curve fits the training data well, at least visually.

Fig. 7: The model is getting closer and closer to the training data

Convergence

This raises the question: When do we consider the training to be complete? The training loop does not have an automatic end; instead, an explicit criterion is needed to terminate the training.

In practice, this is trickier than it seems at first glance. Let’s first take a look at how the MSE develops over the training epochs (Table 1).

The first thing that stands out is that the error decreases over the course of training and converges toward a stable value. This is not a given for more complicated problems, such as if the model does not properly fit the problem or the feedback mechanism for parameter values is too coarse. Achieving convergence during training is a major milestone for many ML projects.

Table 1: Development of the MSE with iterations[.caption}]

Secondly, the MSE becomes small, but it converges to a value greater than zero. This is expected and a good thing, because the model is a simplification of reality. Real data always has some form of noise; for example, even good weather forecasts cannot predict the temperature to two decimal places. If the error becomes too small when training a model, this is suspicious and may indicate that the model has too many parameters and is also mapping the noise in the training data (overfitting). In our example, however, the MSE threshold fits our understanding of the (fictional) domain.

If we knew in advance that the MSE would converge to 0.00003794, we could specify a slightly larger value as the termination criterion. But in practice, we generally don’t know in advance how accurate the model predictions will be, so such an absolute threshold value is out of the question.

Training is often terminated when the relative change in the MSE becomes small enough, e.g., when it changes by less than 0.0001 percent over 100 epochs. This value must also be adapted to the problem at hand: a criterion that is too coarse can terminate training before the possible accuracy is reached, but a criterion that is too fine can prolong training unnecessarily or even lead to an infinite loop due to the finite accuracy of floating-point arithmetic. This is another instance where experience and trial and error are important.

Validation

The fully trained model in our example is ŷ(x) = 0.00041841 + 0.98694384 x + 0.00022286 x² – 0.14406674 x³ and it fits the training data well visually (Fig. 8). But for real-world applications, it’s usually important to know how good the model’s predictions are.

Fig. 8: The converged model fits the training data well from a purely visual perspective

Assuming that the data is normally distributed (often a plausible approximation, which is beyond the scope of this article, see here), the standard deviation is the root of the MSE, i.e., σ = 0.0062 in our example. This means that for two-thirds of the training data, the true values are within ±0.0062 of the model value.

That doesn’t automatically make it good enough. It depends on the domain and context, and assessment is a technical decision. If the accuracy is lower (or higher!) than is technically plausible, this is cause for reflection. You can use a different model but perhaps you’ve learned something new about the domain from the data. It’s important to check the accuracy for plausibility.

So far, we’ve looked at the accuracy of the model on the data we used to train it. It’s also important to consider how well the model can predict values for new, unknown inputs. After all, these predictions are the reason we do ML in the first place.

To do this, you can split the available data and use 80 percent for training, reserving the other 20 percent for subsequent validation. Fortunately, when we did this we only used 200 of the 250 available data points for training. It is crucial that the validation data is never used for training in any way.

The 50 retained data points are plotted as green dots in Figure 9 and visually fit well with the model’s predictions. We verify this quantitatively by comparing the accuracy of the model for training and validation data. The MSE on the training data is 3.79 10-5, on the validation data it is 3.71 10-5, the two values are of the same order of magnitude and it is plausible that the training data is representative and that the model will also fit unknown data.

Fig. 9: Separate validation data can be used to check the model quality

Figure 10 illustrates how a parameter set can be implausible despite good convergence. Here, the same model has been trained with only four data points and has an MSE of less than 10-12, meaning that the prediction accuracy on the training data is phenomenal. Apart from the fact that this is obviously too little training data for meaningful fitting, it only covers part of the value range of x, which is easy to overlook in this diagram.

Fig. 10: Too little or unrepresentative training data can lead to overfitting

During the initial validation step, it may become apparent that model deviations from the training data are significantly lower than expected based upon the technically expected noise of the values. In a joint diagram with the entire available data set, it’s obvious that the parameter values are poor (Fig. 11).

Fig. 11: In this extreme case, the mismatch is obvious when compared to the entire training data set

But this effect can also be much more subtle. Figure 12 shows a curve representing the predictions of a model that was trained with the six points marked in green. The MSE on the training data is 1.6 10-5, which is within a plausible range. The result also seems plausible when looking at the plot.

Fig. 12: The trained model appears to be a good fit for the data as a whole

But if you calculate the prediction accuracy on the validation data, you get an MSE of 1.4 10-4. This is an entire order of magnitude higher than on the training data and is a strong indication that something went wrong during training. That doesn’t just mean that the model has slightly poorer accuracy, it calls the whole process into question: both values should have been the same, but they weren’t, so something must have fundamentally gone wrong somewhere. Conceptual debugging is necessary.

Autograd

Now we’ve walked through how ML model training using an example, from choosing a model to validating the result. We only omitted details about adjusting the parameters to minimize the error. Now, let’s make up for that.

This section is a little more mathematical than the rest of the article, but you can certainly do ML without getting into this level of detail. However, I think it’s good to understand all the steps.

First, we select a parameter, a, and a single data point (x, y). Then we have the question of optimization: How much and in which direction should we change a so that the square error SE at this point becomes smaller?

As a reminder, here are the formulas again:

ŷ = aꞏx³ + bꞏx² + cꞏx + d SE = (ŷ – y)²

So far, we’ve considered these quantities as functions of x, ŷ(x) = …. But we can just as easily consider them as functions of a, without changing the formulas: ŷ(a) = a x³ + … and SE(a) = (ŷ(a) – y)² . We now treat a as the variable and x as a parameter like any other. This is purely a change of perspective, without us having carried out any further analysis, but it is a first step towards investigating how a influences the error.

Figure 13 shows an example of a section of such a function SE(a) with fixed values for x, b, c, and d. For each value of a, the slope of the function gives an indication of the direction and magnitude of the change in a required to get closer to the minimum of SE. If this reminds you of derivatives, that isn’t a coincidence.

Fig. 13: Example excerpt from a curve SE(a) – depending on the position, a must be selected larger or smaller in order to get closer to the minimum

To do this, we need the slope of SE at point a (for the current parameter values). An easy way is to approximate the slope by calculating SE for a second, closely adjacent value of a and dividing the deltas (“difference quotient”) (Fig. 14).

Fig. 14: The difference quotient of error and parameter is an approximation for the differential quotient

The advantage of this brute force approach is that you don’t need to know anything about the underlying function. The disadvantage is that you have to calculate the entire function a second time – and for the slope depending on b another time, for c yet another time, and so on. This is very expensive for large models with thousands or millions of parameters. But it is a potential approach that can benefit greatly from the parallelism of graphics cards.

Many ML frameworks like PyTorch and TensorFlow take a different, often much more efficient approach: they remember the arithmetic operations when calculating the error and differentiate this function symbolically to determine the gradients as a function of the various parameters (autograd).

Figure 15 shows this for our example. The derivative of SE with respect to a is, according to the chain rule, the derivative of SE with respect to ŷ multiplied by the derivative of ŷ with respect to a. The former is 2 (ŷ – y), the latter is x³, so that the total term is 2 (ŷ – y) ꞏ x³.

Fig. 15: Derivation of the error according to parameter a using the chain rule

This calculation is much more favorable than that of the entire model, and it allows for a number of optimizations. For example, ŷ (or even the difference ŷ-y) has already been calculated for the determination of the error and can be reused from there, and x³ is constant across all epochs.

Similarly, the dependence of the error on parameters b, c, and d can be calculated. Averaging these values across all training data yields the parameter corrections for the next epoch.

Conclusion

This artice on machine learning basics used a simple example to show how step-by-step optimization of model parameters can converge over many epochs. It highlighted several fundamental challenges and introduced some statistical tools for recognizing and handling them.

The algorithms and statistical concepts used in this simple example are largely the same as those used for large and complex models and AIs. The rest of the series will pick up on this.

My main concern in this first part was to remove the quasi-magical aura surrounding machine learning. The mathematics and algorithms are not overly complex and basic validation procedures apply on both a large and small scale.