Cross-validation
3- Generalization
Generalization is a frequent term that is used in conversations about Machine Learning. It refers to how able the model can adapt to new, unseen data and how it works effectively using various inputs. It is understandable to say that if new unseen data is inputted into a model, if this unseen data has similar characteristics to the training data, it will perform well [1].
Generalizing things is easy to us humans, however, it can be challenging to Machine Learning models. This is where Cross-Validation comes into the picture [1].
4 . Cross-validation motivation
if you're computing R squared on your test set, the R squared returned is dependent on the way that you split up the data! The data points in the test set may have some peculiarities that mean the R squared computed on it is not representative of the model's ability to generalize to unseen data. To combat this dependence on what is essentially an arbitrary split, we use a technique called cross-validation.
You may see cross-validation also being referred to as rotation estimation and/or out-of-sample testing. The overall aim of Cross-Validation is to use it as a tool to evaluate machine learning models, by training a number of models on different subsets of the input data. Cross-validation can be used to detect overfitting in a model which infers that the model is not effectively generalizing patterns and similarities in the newly inputted data [1].
A typical Cross-Validation workflow
In order to perform cross-validation, the following steps are typically taken:
- Split the dataset into training data and test data
- The parameters will undergo a Cross-Validation test to see which are the best parameters to select.
- These parameters will then be implemented into the model for retraining
- The final evaluation will occur and this will depend if the cycle has to go again, depending on the accuracy and the level of generalization that the model performs.
6- K-fold Cross-Validation Process:
- Choose your k-value
- Split the dataset into the number of k folds.
- Start off with using your k-1 fold as the test dataset and the remaining folds as the training dataset
- Train the model on the training dataset and validate it on the test dataset
- Save the validation score
- Repeat steps 3 – 5, but changing the value of your k test dataset. So we chose k-1 as our test dataset for the first round, we then move onto k-2 as the test dataset for the next round.
- By the end of it you would have validated the model on every fold that you have.
- Average the results that were produced in step 5 to summarize the skill of the model.
import numpy as np from sklearn.model_selection import KFold X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]]) y = np.array([1, 2, 3, 4]) kf = KFold(n_splits=2) for train_index, test_index in kf.split(X): print("TRAIN:", train_index, "TEST:", test_index) X_train, X_test = X[train_index], X[test_index] y_train, y_test = y[train_index], y[test_index]
7. Cross-validation and model performance
As we split the dataset into five folds, we call this process 5-fold cross validation. If you use 10 folds, it is called 10-fold cross validation. More generally, if you use k folds, it is called k-fold cross validation or k-fold CV. There is, however, a trade-off as using more folds is more computationally expensive. This is because you are fittings and predicting more times. This method avoids the problem of your metric of choice being dependent on the train test split. K-fold Cross-Validation is when the dataset is split into a K number of folds and is used to evaluate the model's ability when given new data. K refers to the number of groups the data sample is split into. For example, if you see that the k-value is 5, we can call this a 5-fold cross-validation. Each fold is used as a testing set at one point in the process [1].
8. Cross-validation in scikit-learn
To perform k-fold CV in scikit-learn, we first import cross val score from sklearn dot model selection. As always, we instantiate our model, in this case, a regressor. We then call cross val score with the regressor, the feature data, and the target data as the first three positional arguments. We also specify the number of folds with the keyword argument, cv. This returns an array of cross-validation scores, which we assign to cv results. The length of the array is the number of folds utilized. Note that the score reported is R squared, as this is the default score for linear regression. . We print the scores here. We can also, for example, compute the mean, which we also do.
In [1]: from sklearn.model_selection import cross_val_score
In [2]: reg = linear_model.LinearRegression()
In [3]: cv_results = cross_val_score(reg, X, y, cv=5)
In [4]: print(cv_results)
[ 0.63919994 0.71386698 0.58702344 0.07923081 -0.25294154]
In [5]: np.mean(cv_results)
Out[5]: 0.35327592439587058
9- Why is 10 Such a Desirable k value?
Choosing the right k-value is important as it can affect your level of accuracy, variance, bias and cause you to misinterpreted the overall performance of your model. The simplest way to choose your k value is by equating it to your ‘n’ value, also known as leave-one-out cross-validation. n represents the size of the dataset which allows for each test sample the opportunity to be used in the test data or hold out data.
However, through various experimentations from Data Scientists, Machine Learning Engineers, and Researchers they have found that choosing a k-value of 10 has proven to provide a low bias and a modest variance. Kuhn & Johnson spoke about their choice of k value in their book Applied Predictive Modeling.
The choice of k is usually 5 or 10, but there is no formal rule. As k gets larger, the difference in size between the training set and the resampling subsets gets smaller. As this difference decreases, the bias of the technique becomes smaller (i.e., the bias is smaller for k=10 than k= 5). In this context, the bias is the difference between the estimated and true values of performance”
Let’s take this context and put it in an example:
Let's say we have a data set where N = 100
- If we chose our k value = 2, our subset size = 50 and the difference = 50
- If we chose our k value k = 4, our subset size = 75 and the difference = 25
- If we chose our k value = 10, our subset size = 90 and the difference = 10
So, therefore, as the value of k increases, the difference between the original data set and the cross-validation subsets becomes smaller.
It is also stated that choosing k= 10 is more computationally efficient, as the larger the values of k gets it becomes more computationally impractical. Small values will also be deemed as computationally efficient, however, they pose the possibility of high bias.
Types of Cross-Validation:
- Leave One Out Cross-Validation (LOOCV)
Leave One Out Cross-Validation is a special case of cross-validation technique, instead of creating two subsets, it selects a single observation as test data and the rest of the data as the training data.
This cross-validation happened N number of times where N is the total number of observations. The main disadvantage of LOOCV is that as we have many records like millions of records then it takes higher computation time.
- Hold Out Method
Hold Out Method is the most basic of the cross-validation techniques. It simply divides the dataset into training and testing sets. The training dataset is used to train the model and then the testing dataset is fitted in the trained model to make predictions. This method is used as it is computationally less costly.
4. Stratified Cross-Validation
Stratified cross-validation is the same as the K-Fold cross-validation technique. To tackle the disadvantage of k-fold cross-validation stratified cross-validation comes into the picture. In every iteration, it handles the number of classes to be present in the training dataset.
5. Time Series Cross-Validation
In time series cross-validation, we cannot split our dataset into training and testing datasets. Time series cross-validation starts with a small subset of data for training and makes a prediction for the future data points and then checking the accuracy for the predicted data points.
After that, the same predicted data points are then included as part of the next training dataset and future data points are predicted. Similarly, the process continues in time series cross-validation [2].
