The Tuning Stage

1. Model Complexity

Let’s remember Linear Regression:

\[Y = \beta_0 + \beta_1 X_1 + \epsilon\]

❓What about non-linear behavior as below?

👉 If we add a new transformed feature \(X_1^2\) we have a better fit (in red)

\[Y = \beta_0 + \beta_1 X_1 + \beta_2 X_1^2 + ... + \epsilon\]

🤔 We could engineer very complex features: \(X^3, X^6, (X_1^2 * X_2^5), ...\)

Our \(R^2\) will keep increasing with every additional feature!

We just solved Data Science!

def train_best_model_ever(X,y):
    while True:
        model = LinearRegression()
        model.fit(X, y)
        if calculate_r2(model, X, y) < 0.999999:
            X = add_more_crazy_features(X)
        else:
            return model

Overfitting and Underfitting

The Bias-Variance Tradeoff

One of the most important concepts in Data Science!

Conceptual Definition

Error due to Bias: The error due to bias is taken as the difference between the expected (or average) prediction of our model and the correct value which we are trying to predict.

Error due to Variance: The error due to variance is taken as the variability of a model prediction for a given data point. Again, imagine you can repeat the entire model building process multiple times. The variance is how much the predictions for a given point vary between different realizations of the model.

An optimal balance of bias and variance leads to a model that is neither overfit nor underfit.

📚 Great read

❓How do we find the optimal model complexity?

Solutions for Overfitting

2. Regularization

Regularization means adding a penalty term to the Loss that increases with \(\beta\)

\[Regularized Loss = Loss(X,y, \color{blue}{\beta})\ + Penalty(\color{blue}{\beta})\]

👉 Penalizes large values for \(\beta_i\)
👉 Forces model to shrink certain coefficients or even select less features
👉 Prevents overfitting

The two most famous Regularization penalties are:

\[Ridge(\color{green}{X},\color{magenta}{y},\color{blue}{\beta}) = \underbrace{\sum_{i=0}^{n-1}(\color{magenta}{y_i}-(\color{blue}{\beta_0} + \color{blue}{\beta_1} \color{green}{x_1} + ... \color{blue}{\beta_p} \color{green}{x_p} ))^2}_{Loss(\color{green}{X},\color{magenta}{y},\color{blue}{\beta})} + \color{red}{\alpha} \sum_{\color{green}{j=1}}^{\color{green}{p}} \color{blue}{\beta_j}^{2}\]

\[Lasso(\color{green}{X},\color{magenta}{y},\color{blue}{\beta}) = \underbrace{\sum_{i=0}^{n-1}(\color{magenta}{y_i}-(\color{blue}{\beta_0} + \color{blue}{\beta_1} \color{green}{x_1} + ... \color{blue}{\beta_p} \color{green}{x_p} ))^2}_{Loss(\color{green}{X},\color{magenta}{y},\color{blue}{\beta})} + \color{red}{\alpha} \sum_{\color{green}{j=1}}^{\color{green}{p}} |\color{blue}{\beta_j}|\]

from sklearn.linear_model import Ridge, Lasso, LinearRegression

linreg = LinearRegression().fit(X, y)
ridge = Ridge(alpha=0.2).fit(X, y)
lasso = Lasso(alpha=0.2).fit(X, y)

coefs = pd.DataFrame({
    "coef_linreg": pd.Series(linreg.coef_, index = X.columns),
    "coef_ridge": pd.Series(ridge.coef_, index = X.columns),
    "coef_lasso": pd.Series(lasso.coef_, index= X.columns)})\

coefs\
    .applymap(lambda x: int(x))\
    .style.applymap(lambda x: 'color: red' if x == 0 else 'color: black')

❗️ Warning ❗️

When regularizing a Regression with a L2 penalty or with a L1 penalty:

• We regularize \(\color{blue}{\beta_\color{green}{1}}, ..., \color{blue}{\beta_{\color{green}{p}}}\)

• We do NOT regularize the intercept \(\color{blue}{\beta_0}\)

  • Keep in mind that Ridge reduces the influence of features with a non-significant coefficient

  • Keep in mind that Lasso shrinks them to zero

  • But \(\color{blue}{\beta_0}\) is not associated with any feature!

New hyper-parameter \(\color{red}{\alpha}\)

• Dictates how much the model is regularized

• Large \(\color{red}{\alpha}\) values force model complexity to decrease 👉 ⤵ variance, ⤴ bias

The sum starts from j = 1, we do not penalize the intercept coefficient

ElasticNet = Lasso & Ridge Weighted Average

\[L=\|y-\hat{y} \|^{2}+ \color{red}{\alpha} (\color{orange}{\lambda}|\beta |+(1 - \color{orange}{\lambda})\|\beta \|^2)\]

☝️ Regularization tends to penalize features that are not statistically significant but DO NOT use p-values (do not even look at them). They cannot be correctly interpreted in this case.

ML vs stats

At this stage we do not care anymore about p-values. The fundamental difference between classic stats and ML is that in the former we were trying to infer something from the data, in the latter we are trying to predict something in the most precise and efficient way.

Conclusions

👉 Regularize when you think you are overfitting (e.g. Learning Curves not converging)

👉 Ridge when you believe all coefficients may have an impact

👉 Lasso as a feature selection tool (much better for interpretability!)

✅ Regularization is almost always appropriate

• Ridge is often turned on by default in most Machine Learning models

• You just have to tune the regularization parameter

3. Model Tuning

❓ How to choose the best hyper-parameters (e.g: alpha)

The Grid Search Method

Explores different hyperparameter value combinations to find the combination which optimizes performance

1. Hold out a validation set (never use the test set for model tuning!)

2. Select which grid of values of hyper-parameters to try out

3. For each combination of values, measure your performance on the validation set

4. Select hyper-parameters that produce the best performance

💻 Let’s (manually) fine-tune a linear model with ElasticNet regularization

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.20, random_state=1)

🔥 Grid Search CV

1. Randomy split your training set into k folds of same size

2. Make fold #1 a val_set, train model on other k-1 folds & mesure val_score

3. Make fold #2 a val_set and repeat

4. …

5. Compute average val_score over all folds

• Repeat for each value of hyper-param to test

• Save the test set for final evaluation only (AFTER hyper-params are chosen)

Welcome to Sklearn GridSearchCV 🚀

from sklearn.model_selection import GridSearchCV

# Train/Test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=1)

# Instantiate model
model = ElasticNet()

# Hyperparameter Grid
grid = {
    'alpha': [0.01, 0.1, 1], 
    'l1_ratio': [0.2, 0.5, 0.8]
}

# Instantiate Grid Search
search = GridSearchCV(
    model,
    grid, 
    scoring = 'r2',
    cv = 5,
    n_jobs=-1 # parallelize computation
) 

# Fit data to Grid Search
search.fit(X_train, y_train);

👎 Limitations of Grid Search:

Random Search

Randomly explore hyperparameter values from:

• A hyperparameter space to randomly sample from

• The specified number of samples to be tested

Sklearn’s RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV
from scipy import stats

# Instantiate model
model = ElasticNet()

# Hyperparameter Grid
grid = {'l1_ratio': stats.uniform(0, 1), 'alpha': [0.001, 0.01, 0.1, 1]}

# Instantiate Grid Search
search = RandomizedSearchCV(
    model,
    grid, 
    scoring='r2',
    n_iter=100,  # number of draws
    cv=5, n_jobs=-1
)

# Fit data to Grid Search
search.fit(X_train, y_train)
search.best_estimator_

Choose hyperparameter probability distribution:

from scipy import stats

plt.figure(figsize=(5, 3))

dist = stats.norm(10, 2) # if you have a best guess (say: 10)

dist = stats.randint(1,100) # if you have no idea
dist = stats.uniform(1, 100) # same

dist = stats.loguniform(0.01, 1) # Coarse grain search

r = dist.rvs(size=10000) # Random draws
plt.hist(r);

RandomizedSearch vs GridSearch

👉 Randomized Search:

• Less typing, if you want to try many values
• Control for the number of combinations to try & time spent searching
• Useful when some hyper-parameters are more important than others

In any case:
✅ Always start with a coarse grain approach (can use Grid or RandomSearch)
✅ Then afterward, fine-tune your search

Bayes theorem provides a framework that can be used to quantify the beliefs about the “location” of the optimal parameter set in the search-space.

Tuning hyperparameters using Bayesian logic—helps reduce the time required to obtain an optimal parameter set.

📚 Great read

HyperOpt

pip install hyperopt

Objective function

# Define the objective function to minimize (negative mean cross-validated R^2 score)
def objective(params):
    # KNN regressor with hyperparameters to optimize
    knn = KNeighborsRegressor(
        n_neighbors=params['n_neighbors'])

    # Cross-validated R^2 score
    score = np.mean(cross_val_score(knn, X_train_scaled, y_train, cv=5, scoring='r2'))
    
    # We want to maximize R^2, so we minimize the negative score
    return -score

Search Space

# Define parameter space using hyperopt random variables
# Define the hyperparameter search space
space = {
    'n_neighbors': hp.choice('n_neighbors', np.arange(1, 11,1))  # Number of neighbors
    #"n_neighbors": scope.int(hp.quniform("n_neighbors", 2, 30, 1)),

}

# Set up trials for tracking
trials = Trials()

# Use the Tree Parzen Estimator (TPE) algorithm for optimization
best = fmin(fn=objective, space=space, algo=tpe.suggest, max_evals=50,
            trials=trials,
            rstate=np.random.default_rng(123)) # fixing random state for the reproducibility)

5. PYCARET

PyCaret is a simple, easy to learn, low-code machine learning library in Python. With PyCaret, you spend less time coding and more time on analysis.

pip install pycaret