In this article, we looked at the mathematical foundations of machine learning (ML): What is a model, and how do you train its parameters so that the model makes the best possible predictions? Now, let’s take a step back to examine practical questions that arise when trying to solve real-world problems with ML.

Previously, we trained a simple model. The goal was to predict a value y (dependent variable) based on the value of a variable x (independent variable).

We decided to use a third-degree polynomial as our model: ŷ(x) = a x³ + b x² + c x + d. The model has four parameters—a, b, c, and d—which we adjusted iteratively using numerical methods so that the model curve fits the training data (Fig. 1).

Fig. 1: Training data with the trained model

Then we validated the model’s predictive quality using separate validation data that wasn’t used for training. We used a polynomial as the model at our discretion, but we could just as easily have used a different function with different parameters—more on that later. First, let’s look at some details about the training iterations.

Learning Rate

The model is trained in a loop (Fig. 2). The process begins by applying the model to the training data and calculating the prediction error.

Fig. 2: Training loop

This is used to determine which parameters need to be adjusted, by how much, and in which direction, in order to reduce the error (the gradient of the parameters). This is achieved with the autograd algorithm. The gradient is subtracted from the parameters, and the next iteration of the loop uses this—hopefully improved—set of parameters.

This final step of calculating the parameter’s gradients raises a practical problem. The gradient indicates the “direction” in which the model must change to improve—that is, which parameters need to be increased or decreased and by how much. However, the gradient says nothing about how far the model is still from the optimal point, i.e., what the best weighting factor for the gradient is. This factor is called the learning rate and is often abbreviated with the Greek letter η: Pn+1 = Pn – η grad.

A learning rate that’s too small will improve the model slowly in the early stages, even though much needs to change and large steps will suffice. This drives up training effort immensely, easily by several orders of magnitude in practice.

But a learning rate that’s too high limits the accuracy that the parameters can be adjusted to the training data. In extreme cases, learning rates that are too high can cause the model to “oscillate” without finding the optimum, preventing the training loop from converging at all.

In practice, algorithmically adjusted learning rates are almost always used, especially ADAM (Adaptive Moment Estimation). ADAM estimates from the gradient’s behavior how much “momentum” the changes currently have, and reduces the learning rate when the changes become smaller or increases it when the rate of change picks up.

Batching

The first step of the training loop—applying the current model to the training data—can also be significantly optimized compared to a naive approach.

As we recall, the goal of training is to determine a set of parameters that minimizes the mean error when we apply the model to all the training data. To achieve this, at the start of each iteration we applied the model to all training data and calculated the mean error. As a result, the more training data we have, the longer each iteration takes. If you have a lot of training data, this can become prohibitively expensive. More training data means (assuming good data quality) statistically better, more accurate results. But if you assume a fixed time or CPU budget for training, a hundred times more training data means you can only perform one-hundredth as many iterations. Ultimately, that can result in a poorly trained model.

Of course, you can reduce training time by using only a portion of the available training data. But this isn’t satisfactory and compromises the quality of results. Selecting a subset of the training data is more difficult than it seems at first glance. The selection must be representative; otherwise, you introduce a systematic error (bias). For example, if you overrepresent certain stores and underrepresent others in data on shopping behavior, it can influence the results in unwanted ways. But if you have an extremely large amount of training data, a selection is still necessary.

There’s a middle ground for training with large amounts of data without drastically increasing the number of epochs: batching. To do this, divide the training data into subsets (batches) of typically a few hundred or thousand data points and run the training loop through them one after another.

In the first pass, the model parameters are adjusted based on the data from the first batch. With these adjusted parameters, the second iteration starts on the second batch, and so on. If we have, say, 20 batches, the 21st pass starts again on the first batch.

In each individual iteration, adjusting the parameters moves roughly in the right direction. It isn’t quite as good as it would be if it was based on the entire training dataset, but it comes at a fraction of the effort. And over the course of the iterations, all of the training data is incorporated into the result. In practice, batching often yields results that are good enough and this approach is widely used.

When dividing the training data, it is extremely important that each batch is as representative of the entire dataset as possible. For example, if we divide the training data from Figure 2 so that each batch contains a different range of values for x, the individual iterations pull the model in such different directions that it no longer converges. In practice, randomly selecting the batches is usually sufficient to achieve this even distribution.

Choosing the Model

In the first example, we selected a third-degree polynomial ŷ(x) = ax³ + bx² + cx + d with four parameters as our model. During training, we adjusted the parameters so that the model fits the data.

But why a third-degree polynomial and not, for example, a deep neural network or a fifth-degree polynomial? This is a technical decision that we make as humans; there is no objectively right or wrong answer.

The selection process takes into account an understanding of the subject domain. Other requirements like simplicity or interpretability also play a role. In practice, we often try out different models to see which one works best.

At first glance, the training data in our example appears point-symmetric around the origin, so an odd-degree polynomial fits better than an even-degree one. We could add a term e^x⁵ and train the model with it—whether this significantly improves prediction accuracy can only be determined by trial and error. In our case, a third-degree polynomial provides the necessary accuracy and has the advantage of being simpler and more efficient.

Based on our domain expertise, if we know that the data is actually point-symmetric and deviations are merely noise, then we can omit the linear terms from the model and train the model ŷ(x) = a x³ + c x. This model approximates the training data only slightly worse than the model with linear terms; the parameters b and d in the fully trained model were small but not zero. But based on our domain knowledge, we know that it’s a better fit and training the linear terms would only approximate noise in the training data.

But with a linear polynomial ŷ(x) = bx² + d as the model, training the model on the training data is impossible. There’s simply no quadratic function that even remotely approximates the data.

This is a simple example of how choosing a model requires experience, domain knowledge, and trial and error. In practice, there’s a wealth of potential models. Some work better, some worse, and most don’t yield meaningful results at all. Tutorials for ML libraries often overlook this point; they present a well-suited model as if it were a given and focuses on the mechanics of training.

If we go one step deeper into the fictional domain from our example, we may find that the data describes a harmonic oscillation. This can be described using the model ŷ(x) = a sin(ωx + φ), with amplitude a, angular frequency ω, and phase shift φ as parameters.

This model has the advantage of being an accurate representation of the domain. It is also simple, requiring only three parameters, and interpretable. The parameters a, ω, and φ have domain-specific meanings. The trained model makes sense and can be understood by domain experts.

Whether and to what extent these characteristics are important depends on the specific case. But it’s good to keep them in mind. Neural networks aren’t the best model for every problem; domain expertise should drive the decision.

Input Parameters

Domain knowledge can also be applied when selecting input parameters. As an example, let’s consider a classic neural network problem: the cart with a pivoting vertical pole. In this problem, the model must control the cart’s movement so that the pole does not fall over (Fig. 3).

Fig. 3: Cart with a rotating pole

I tried to code the example myself, using the car’s position and the angle of the rod over the last ten laps as input. I couldn’t manage to get even a remotely functional result.

Then I looked at the original paper, which is already over 40 years old. The authors don’t use the history of positions as input; instead, they calculate the angular velocity of the rod and the horizontal velocity of the cart outside the model and pass these pre-calculated values to the model. They do not pass the tilt angle as a number; instead, they define angle ranges and pass flags indicating in which angle range the rod is straight.

At first, this felt like cheating to me. After all, a good model should be able to derive results from the raw data, right? But in reality, it makes perfect sense to select and prepare the input data so that the model and training algorithm work as well as possible.

Of course, this requires experience and a willingness to experiment. The article about this model presents the parameter choice as if it was obvious and describes only the good results the authors achieved. But in practice, choosing good input parameters requires a lot of hard work and trial and error, even if publications make it seem otherwise.

Outer Loop

In this article, we’ve examined a number of adjustment parameters to improve the model, make it simpler, and simplify training: learning rate, batching, choice of model, and input parameters.

These adjustment parameters are also called hyperparameters: settings that are not part of the model but still influence the results. Just as the training loop optimizes the model’s parameters, there is a second, outer loop for optimizing the hyperparameters.

Figure 4 illustrates this schematically. You train a model with the current selection of hyperparameters, evaluate the quality of the results, adjust the hyperparameters, and train the new model. Unlike model training, where parameter adjustment occurs algorithmically and automatically, optimizations in this outer loop are based on human experience.

Fig. 4: Training loop for hyperparameters

This concludes the second part of this series and our introduction to tuning parameters and practical approaches.