[question] What’s the point with regularization in the case of linear models

[945fc41467]

Enter the chapter number

4.5

Enter the page number

257

What is the cell's number in the notebook

No response

Enter the environment you are using to run the notebook

None

Question

Hello,

A little question about regularization.

Non parametric models, like random forests, make no hypothesis on the distribution of the data and can adapt to any shapes. The counterpart is they can even fit to the noise and local errors, which lead to overfitting. In this context, I perfectly understand the need of regularization : we force the model to not adapt to much to the data, to prevent overfitting.

But in the case of a parametric model with few parameters, like a linear models, I don’t understand what’s the point with lasso, ridge, elastic net regression, etc. It can be proved that minimizing the squared sum give the best unbiased estimators for all parameters with the minimum variance. Is there a mathematical justification to add a penalty to this natural loss function ? Or, if it is totally empirical, why this difference between theory and practice ?

Thanks

[ageron]

Great question!

Under classical linear model assumptions, you are right that ordinary least squares (OLS) is best. However, these assumptions rarely hold perfectly. In particular, if some features are highly correlated, then X^TX is ill-conditioned or nearly singular. In this case, OLS estimates have a very high variance (i.e., tiny perturbations can cause huge swings in the result). Ridge regression stabilizes this, even when using a fairly small alpha.

Maybe I should add this point to the book, actually. Thanks for asking!

[ageron]

Also, even if the data isn’t linear, OLS will fit the best linear approximation. If the data is noisy, then regularization can help avoid fitting the noise within that linear approximation.
For example, suppose the data is y = max(2x + Gaussian noise, 0), with x ≥ 0. The noise is not symmetrical, since it is bounded at 0. This means that OLS will overestimate the slope.

[ageron]

I just added a paragraph about this in the upcoming book, thanks again for your excellent question.

[945fc41467]

Thanks for your answer, and sorry for the late reply.

So, if I understand well, there is no reason to use regularization if hypothesis are perfectly respected ?

And why this regularization should only happen in the direction of lowering parameters ? In the second example you give, it seems to me that the slope would be underestimate : close to zero, negative values are increased to zero, leading to a higher mean that it should, and far from zero the mean is unchanged. So, the intersect should be overestimate but the slope should be underestimate. Am I wrong ?

If I think about your first example (correlated features) in term of projection :

• some vectors features points almost in the same direction. So small changes in this direction of one of this vector make a great change in the hyperplan defined by the features when we are fare from the main axis of these features.
• regularization consist in using a projected vector shorter and closer to the features vectors, making residuals bigger, but projected vector less sensitive to variations in the hyperplan orientation. This example make more sense to me

[945fc41467]

Another though : for the case of highly correlated feature, a PCA before applying the linear model wouldn’t be simpler and more rigorous than applying a regularization ?

[purushottamk3112]

Great question!

Under classical linear model assumptions, you are right that ordinary least squares (OLS) is best. However, these assumptions rarely hold perfectly. In particular, if some features are highly correlated, then XTX is ill-conditioned or nearly singular. In this case, OLS estimates have a very high variance (i.e., tiny perturbations can cause huge swings in the result). Ridge regression stabilizes this, even when using a fairly small alpha.

Maybe I should add this point to the book, actually. Thanks for asking!

yes,correct