1 Training/Dev/ Test set Data

1.1  Dev and Test sets must come from the same distribution

Development set (Dev set): it is also called hold out cross-validation set. How to divide data in dev set and test set;  for example take these three changes as dev set and take the other four regions as the test set and it is a very bad idea because in this example dev and test set come from a different distribution. the better idea is that find a way in which dev and test set come from the same distribution.  when all the month spent on optimized the system on dev set and let you find it is not given the good performance on the test set. To avoid this problem you randomly shuffle data into Dev and test set so that both data come from the save the region.

1.2 How large Dev and test set data

According to the role of the thump of ML,  take all the data you have and using 70% and 30%  split into train and test set. if you set dev and test set . you may use 60% train, 20% dev and 20% test. In old ML history, this heuristic is reasonable when dataset size is smaller (1000) but it is not reasonable if the example is 10,000.

But in modern ML;  we used a much larger dataset. For a million data set an example, it might be reasonable that 90% training, 1% set dev and 1% test

1.3 size of the test set

After the development of the system , the purpose of the test set is to check the performance of the system. For some example some time you need high the accuracy or high confidence;  then for such  application, if your dataset size is 10,000 ,1000,000 example then 30% for test set is sufficient. for some application where you do not need high accuracy; for such system only training and dev set is sufficient and you do not need a test set but is not recommended.

2. Bias and Variance

during check the system performance we see the training error and dev error. if train error is 1% and dev error is 10%. if dev and train data come from the same distribution, you said that it is a variance problem and it means that your algorithm is not generalized very well, if the train and dev set to come from different distribution then you cannot draw this conclusion.

let define a new set of data called train-dev set . this is the new subset of data which have the same distribution as train data but not used for training, Now you train your NN on training data. let train error 10%. train-dev error is 9% and the error on dev set is 10%, you still have variance problem.

References