1.2. Model Selection

One of the most important steps in a Machine Learning (ML) pipeline is to select the best available model to deploy in the desired task. This is closely related to hyperparameter tuning (HP-tuning), which is the process of finding the best hyperparameters (HPs) of the model for the given problem. These problems involve estimating the performance of a model on unseen data – data that was not available at training time (Raschka 2018). For clarity, model evaluation is the process of estimating how well a model generalizes on unseen data; model selection is the process of choosing the best hyperparameters for a given model; and Model comparison is the process of comparing the performance of different models.

Holdout Method

The most used method for model evaluation is the holdout method. This involves splitting the dataset \(\mathcal D = \{(\boldsymbol x_i, \boldsymbol y_i)\}_{i=1}^N\) with \(N\) observations into two disjoint splits, called train \(\mathcal D_{\text{Train}}\) and test \(\mathcal D_{\text{Test}}\) split. The train split is used for training the model, while the test set is used only to evaluate the model performance. Generally, the test split contains about \(10-30\%\) of the total amount of data. This method is visually depicted in Figure 1.

Figure 1: Diagram of the holdout method.

The test set effectively simulates the behavior of the model when seeing data that was not in training. If the model generalized well, it should maintain a similar performance as it did on the training split. Note that this assumes that both splits were sampled from the same distribution, therefore, the model was able to learn patterns that are representative for that dataset. If new data is provided to the model, which possesses completely different statistics than the training data, the model is going to perform worse than expected – this is known as the Out-Of Distribution (OOD) problem.

A variant of the holdout method is the stratified holdout method, which is important for classification problems. In this setup, the splits can not be randomly sampled from the original dataset, as the splits need to have the same class frequencies as the original data. For that, stratified holdout makes the sampling obey this condition, which leads to the train and test split presenting approximately the same class proportions as the initial dataset. This is depicted in Figure 2.

Figure 2: Diagram of the stratified holdout method. The darker colors represent the number of observations of the class \(c_1\) and the lighter ones of the class \(c_2\).

The actual metrics used for model evaluation are introduced in Model Evaluation. Note that the Pre-Processing step is always performed after the data splits have been made. When data is normalized using statistics derived from the entire dataset, splitting the observations into training and testing sets leads to data leakage. This occurs because the test set utilizes statistics calculated from the training set, potentially causing biased estimations. The best practice is to first perform the holdout method, then apply pre-processing algorithms – such as feature selection or normalization – using only the training set statistics, and finally apply the resulting transformations to the test set.

\(k\)-fold Cross-Validation

Another sub-problem in an ML pipeline is model selection, which involves choosing the optimal set of HPs for a given model. A naive approach is using the holdout method to select the best-performing model. However, a key issue with this approach is that the performance on the test set may be biased by the specific split performed. Furthermore, there is no separate test set to evaluate model performance on unseen data, as the test set must be reserved for selecting the optimal HPs.

To address these issues, \(k\)-fold Cross-Validation (CV) is employed. First, the method applies the holdout technique to create a training and testing split. Then, using the training set, it generates \(k\) disjoint sets \(\mathcal D_i \subset \mathcal D_{\text{Train}}\), such that: \[ \bigcup_{i=1}^k \mathcal D_i = D_{\text{Train}}. \tag{1}\] Subsequently, the \(k\)-fold CV trains the model multiple times, once for each HP configuration. For each iteration, the model is trained on \(k-1\) sets, using the remaining set as the validation set – analogous to the test set. Finally, the model yields \(k\) performance values, and the mean of these values is used to estimate the model’s true performance. This average performance is then used to select the best possible HP configuration from a predetermined search space.

Note that by utilizing different splits, \(k\)-fold CV effectively reduces split bias. Typically, \(k=10\) is used for this method, although the choice depends on factors such as dataset size, computational cost, and other considerations. Visually, the CV process is depicted in Figure 3.

Figure 3: Diagram of the \(k\)-fold CV with \(k=4\).

Code

Here, a simple Machine Learning pipeline will be conducted using the \(k\)-NN model. Importing packages.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, GridSearchCV, KFold
from sklearn.neighbors import KNeighborsClassifier

Loading the Iris dataset.

iris = load_iris()
X, y = iris.data, iris.target

print(f"Dataset shape: {X.shape}")
print(f"Classes: {iris.target_names}")

Dataset shape: (150, 4)
Classes: ['setosa' 'versicolor' 'virginica']

Performing the train and test split, with 70/30 ratio.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=100)

Creating a hyperparameter search space.

param_grid = {'n_neighbors': [1, 3, 5, 7, 9, 11, 13, 15]}

Setting a \(5\)-fold Cross-Validation.

cv = KFold(n_splits=5, shuffle=True, random_state=100)

Fitting and finding the best value of \(k\).

grid_search = GridSearchCV(
    estimator=KNeighborsClassifier(),
    param_grid=param_grid,
    cv=cv,
    scoring='accuracy'
)

grid_search.fit(X_train, y_train)

print(f"Best k: {grid_search.best_params_['n_neighbors']}")
print(f"Validation Accuracy: {grid_search.best_score_:.4f}")

Best k: 3
Validation Accuracy: 0.9714

Evaluating in the test set.

best_knn = grid_search.best_estimator_
final_test_accuracy = best_knn.score(X_test, y_test)

print(f"Test Accuracy: {final_test_accuracy:.4f}")

Test Accuracy: 0.9778

Source: model_selection.ipynb

References

Raschka, Sebastian. 2018. “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.” arXiv Preprint arXiv:1811.12808.