Loss function :

Loss function is what SGD is attempting to minimize by continuingly updating the weights of the model during training. 

It’s a method of evaluating how well specific algorithm models the given data. If predictions deviates too much from actual results, loss function would cough up a very large number. Gradually, with the help of some optimization function, loss function learns to reduce the error in prediction.

Common loss function 

Mean Squired error: MSE

Categories of Loss functions

Classification losses

In classification, we are trying to predict output from set of finite categorical values i.e Given large data set of images of hand written digits, categorizing them into one of 0–9 digits

• the average of squared difference between predictions and actual observations
• only concerned with the average magnitude of error irrespective of their direction
  
• It is sensitive to outliers 

• absolute differences between predictions and actual observations
• MAE is more robust to outliers since it does not make use of square
• It is not sensitive to outliers 
• 
  

• This is same as MSE with the only difference that we don’t take absolute values

 Regression losses 

Regression, on the other hand, deals with predicting a continuous value for example given floor area, number of rooms, size of rooms, predict the price of room