How to Detect Overfitting and Underfitting in scikit-learn

In machine learning, training a model isn’t just about getting high accuracy. It’s about getting the right accuracy — that generalizes well to new data.

This is where overfitting and underfitting come into play.

Machine learning models can never make perfect predictions: the test error is never exactly zero. This failure comes from a fundamental trade-off between modeling flexibility and the limited size of the training dataset.

The first presentation will define those problems and characterize how and why they arise.

Then we will present a methodology to quantify those problems by contrasting the train error with the test error for various choice of the model family, model parameters. More importantly, we will emphasize the impact of the size of the training set on this trade-off.

Finally we will relate overfitting and underfitting to the concepts of statistical variance and bias.

In this guide, you'll learn how to detect if your model is overfitting or underfitting using scikit-learn, with practical Python code examples and explanations.

🧠 What Are Overfitting and Underfitting?

✅ Strategy to Detect Overfitting / Underfitting

The simplest way:
👉 Compare accuracy (or loss) on training and test sets.

🔁 Use train_test_split() from scikit-learn

🔬 Python Example Using Decision Tree

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Load data
X, y = load_iris(return_X_y=True)
# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Train the model
model = DecisionTreeClassifier(max_depth=None) # try different depths
model.fit(X_train, y_train)
# Evaluate
train_acc = accuracy_score(y_train, model.predict(X_train))
test_acc = accuracy_score(y_test, model.predict(X_test))
print(f"Training Accuracy: {train_acc:.3f}")
print(f"Test Accuracy: {test_acc:.3f}")

🔍 How to Interpret Results

⚠️ If there’s a big gap between training and test accuracy, your model is likely overfitting.

📈 Visualize with Learning Curves (Optional but Powerful)

Scikit-learn has a built-in tool to plot learning curves — which is very helpful to spot overfitting and underfitting.

from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt
import numpy as np
train_sizes, train_scores, test_scores = learning_curve(
 estimator=model,
 X=X,
 y=y,
 train_sizes=np.linspace(0.1, 1.0, 10),
 cv=5,
 scoring='accuracy'
)
# Calculate mean and std
train_mean = train_scores.mean(axis=1)
test_mean = test_scores.mean(axis=1)
# Plot
plt.plot(train_sizes, train_mean, label="Training score")
plt.plot(train_sizes, test_mean, label="Cross-validation score")
plt.xlabel("Training Set Size")
plt.ylabel("Accuracy")
plt.legend()
plt.title("Learning Curve")
plt.grid()
plt.show()

🔍 Interpretation of the Learning Curve:

• Overfitting: Large gap between training and test curves

• Underfitting: Both curves are low and close together

• Good Fit: Curves are high and close together

🛠 Bonus Tips to Fix Issues

• Fix Overfitting:

  • Reduce model complexity (e.g., lower max_depth in trees)

  • Use regularization (e.g., alpha in Ridge/Lasso)

  • Add more training data

  • Use dropout (in neural networks)

• Fix Underfitting:

  • Use a more complex model

  • Add features or interactions

  • Reduce regularization

🧠 Key Takeaways

• Overfitting and underfitting can be detected by comparing training and test performance

• Use train_test_split() and accuracy scores for a quick check

• Visual tools like learning curves help you diagnose the problem clearly

• Scikit-learn makes it easy to experiment with model settings and cross-validation

References