Train, Test, & Validation Sets
For our model we should have our data broken down into three distinct data
- training set
- a validation set
- a test set
training set the training set is what it sounds like it's the set of data used to train the model during each epoch our model will be trained over and over again on this same data in our training set and it will continue to learn about the features of this data. later we can deploy our model and have it accurately predict on new data that it's never seen before it'll be making these predictions based on what it learned about the training data
the validation set is a set of data separate from the training set that is used to validate our model during training this validation process helps give information that may assist us with adjusting our hyper parameters so recall how we just mentioned that with each epoch during training the model will be trained on the data in the training set well it will also be simultaneously validating on the data in the validation set
During the training process the model will be classifying the output for each input in the training set after this classification occurs the loss will then be calculated and the weights in the model will be adjusted then during the next epoch it will classify the same input again now also during training the model will be classifying each input from the validation set as well it will be doing this classification based only on what
the test set is a set of data that's used to test the model after the model has already been trained this test set is separate from both the training set and validation set after our model has been trained and validated using our training and validation sets we'll then use our model to predict the output of the data in the test set now one major difference between the test set and the other two sets that we've discussed is that the test set should not be labeled the training set and validation set has to be labeled so that we can see the Metrics given during trainings like the loss in the accuracy from each epoch. so when the model is predicting on unlabeled data in our test set this would be the same type of process that would be used if we were to deploy our model into the field