Measuring Model Fit

Here we explore various methods for assessing the learning process. We explore how thinking about the output effects how we measure model fit, loss functions (used during the training process), and metrics (used once training is done).

Once you’ve thought about the output, you should have an idea of what kind of data you want to see as output. Maybe its a Boolean, True/False. Maybe its a string, the next expected word in a sentence. Maybe its a probability density function (PDF), "the chance that the variable will be either 1, 2, or 3". Maybe you have an array of numbers that indicate what the expected stock prices should be. Regardless of what your output is, you need a way to measure the performance of your modeling technique. In the context of machine learning, this means you want to measure how well your model has fit the data.

During training, a loss (also called cost) function is used to optimize the model training. This is how the model judges its own performance during training, updates hyperparameters/tuning parameters in response to the loss function, or even implementing early stopping to prevent overfitting.

A loss function is often times naturally chosen; for instance, if we are classifying images where there is only possible label to apply, we would use categorical cross entropy (also known as "softmax loss" because it is the same as the softmax activation function you will learn more of below, generalized as a loss function) as the loss function.

Keras offers many different loss functions. You can find the full list here (along with detailed explanations on which one should be used for your project) and we’ve listed the current (as of Fall 2023) ones below:

Probabilistic losses

Regression losses

Hinge losses for "maximum-margin" classification

There are many metrics, or ways to measure, model fit once training is done. Some of the most well known include accuracy, mean squared error (MSE), mean, categorical cross entropy, or AUC (Area Under Curve). How you measure performance is contextual to your problem, and often times there are many ways that might apply.

While you certainly can construct your own metrics, most data science toolkits have metrics already available as classes/methods. For instance, check out Pytorch’s list of metrics.

Metrics are often chosen in the context of the output. For instance, in classification, accuracy is an obvious choice: what percentage of images are classified correctly? Maybe we have multi-label output (that is, a given classification might have more than 1 label that applies to it), in which case we might use categorical cross entropy.

The names of metrics can vary between the various toolkits, so sometimes you will have to dig into the docs and read what metrics come with it, and what they do.

Activation functions are used for 2 primary purposes:

They are used especially for neural networks, and are often applied at the end of the training process to produce an output that is guaranteed to produce nonlinear outputs.

Consider one of the most commonly used activation functions for neural nets, ReLU ("Rectified Linear Unit") whose equation is

$ g(z) = (z)_+ = \left\{ \begin{array}{ll} 0 \ \ \ \ \ if \ z<0 \\ z \ \ \ \ \ otherwise \end{array} \right. $

This isn’t the right place to go into detail about this particular activation function, but our source (the Introduction to Statistical Learning (Chapter 10.1) has a much more in depth explanation if you are curious. This will produce a shape like below:

Another function that is commonly used is the sigmoid (sometimes called logistic because it is used for logistic regression) function. The sigmoid function converts linear functions into probabilities between 0 and 1. This is used for binary classification problems ("what is the chance this image has a cat in it") where the output is a binomial probability distribution.

Yet another common function is the softmax function. Softmax is similar to sigmoid, but it differs in that it maps multiple probabilities in the 0 to 1 range. For instance, given an image that might have a dog, cat, or pig in it (so 3 labels) and we know only one of them will be in the image (so this is multi class but not multi label because only 1 label is being applied), our softmax function would return 3 numbers that sum to 1 that represent the odds that the image is a dog [0], cat [1] or pig [2].

Recall that train, valid and test splits differ in their involvement in training a model. Its important to know that often, training metrics will appear the most positive, validation metrics often the second, and testing metrics are often the lowest, assuming they are all not equal.

The reason why they differ has to do with the training process. Recall that the training data is used for training the model, the validation split is used to verify performance and/or optimize the tuning parameters, and the test split has no involvement in training. In theory, if training goes well, all 3 metrics should be the same (and good in the context of the metric, whether that means a high percentage of accuracy, high $R^2$, etc). In practice, this is rarely the case.