Regularization Part 1: Ridge (L2) Regression
Why Cross Validated Stack Exchange recommends it
Cross Validated answerers and commenters point people to it for an intuitive, step-by-step explanation of ridge regression and shrinkage, including the one-predictor and more-predictors-than-samples cases.
Recommended by 4 people across 3 different months, 2020 to 2020.
What it is
A 20-minute StatQuest episode in which Josh Starmer explains ridge (L2) regression: how adding a penalty shrinks coefficients to reduce overfitting, how it works for continuous and discrete predictors and logistic regression, and how it can handle cases with more parameters than data points. It builds on earlier StatQuest videos on bias and variance, linear models and cross validation.
What people said
Recommends Kenneth Enevoldsen · 2020-05 · Cross Validated Stack Exchange · on “Predicting a categorical variable with a bag-of-words matrix that has more columns than rows”
It indeed seems like what Otto is mentioned might be the issue, but I will elaborate a bit more on it. So what you might have and is likely to have is a dimensionality issue, meaning that you have more predictors than data points. This will lead to your model being unable to fit as it can always make a perfect fit. This is also called the 'curse of dimensionality' and have been an issue for a long time in NLP and is still problematic is fields such as genetics and neuroscience. One way to solve this is as Otto mentions using regularizations. E.g. ridge regression or similar. I personally would suggest this video [this video] by Josh Starmer. Alternatively, you might want to shift to a classifier
Recommends Atakan · 2020-05 · Cross Validated Stack Exchange · on “why regularization is slower slope and not higher?”
Josh Starmer's this StatQuest video [this video] helped me understand some of these concepts in a very relatable way. There are multiple follow up videos on Lasso, Ridge and Elastic regression approaches.
Recommends Dave · 2020-07 · Cross Validated Stack Exchange · on “Why can't do ridge regression with one predictor?”
StatQuest does ridge with one predictor just fine in his video. [this video] The method is somewhat silly to use in a regression with just one parameter, but I am surprised the common software implementation don’t allow it. Perhaps the StatQuest example could make sense in some setting. But that’s just an issue with the software implementation. You’re still able to write your parameter vector as $\hat{\beta}_R = (X^TX+\lambda I)^{-1}X^Ty$ and do the calculation. ( $I$ is the identity matrix; $\lambda$ is your ridge regression hyperparameter.) Another popular software implementation of ridge regression is the sklearn packing in Python. Perhaps give that a whirl.
Recommends Pace · 2020-08 · Cross Validated Stack Exchange · on “Why does shrinkage work?”
@Kolassa has a great mathematical answer. For a more intuitive visual answer here is a picture. I'm doing simple linear regression here with a slope and y-intercept. A population of 17 points are loosely correlated. At random I picked two points and created a regression. In general, 2 points is not enough observations and my regression lines are going to vary wildly in shape and quality. However, the r^2 error will be perfect, the line hits both of my test points. The solid lines (R1 through R5) represent these regressions. The dashed lines (G1 through G5) represent the regression with a shrinkage effect applied. [LINK] Shrinkage is going to shrink the slope towards zero. This isn't an arbit