L1 vs L2 Weight Regularisation Paths: Lasso Sparsity vs Ridge Weight Decay

When a regression model fits training data too closely, it often fails on new data. Regularisation is the standard fix: we add a penalty that discourages overly complex solutions, improving generalisation and stabilising estimates. If you are studying model tuning in a data scientist course in Kolkata, two penalties come up repeatedly—L1 (Lasso) and L2 (Ridge). Both shrink coefficients, but they do it in very different ways, and the difference becomes especially clear when you look at regularisation paths (how coefficients change as the penalty strength varies).

Why regularisation changes model behaviour

Consider linear regression with features XXX, target yyy, coefficients β\betaβ, and intercept bbb. Ordinary least squares minimises the residual sum of squares:

Minimise: ∑i(yi−(b+Xiβ))2\sum_i (y_i – (b + X_i\beta))^2∑i​(yi​−(b+Xi​β))2

Regularised regression adds a penalty term controlled by λ\lambdaλ (sometimes written as α\alphaα):

• Ridge (L2) adds λ∑jβj2\lambda \sum_j \beta_j^2λ∑j​βj2​
• Lasso (L1) adds λ∑j∣βj∣\lambda \sum_j |\beta_j|λ∑j​∣βj​∣

As λ\lambdaλ increases, the penalty matters more, and coefficients are forced toward smaller values. The key distinction is what happens on the way to smaller values: Lasso tends to create zeros (sparsity), while Ridge tends to create small-but-nonzero coefficients (weight decay).

L1 vs L2: the intuition behind sparsity and shrinkage

A useful mental model is to think of the penalty as defining an “allowed region” for coefficients.

• With L2, the constraint region looks like a smooth circle (or sphere in higher dimensions). Smooth boundaries encourage solutions where coefficients are reduced gradually.
• With L1, the constraint region looks like a diamond (with sharp corners aligned to axes). Those corners make it more likely the optimum lands exactly on an axis—meaning some coefficients become exactly zero.

That geometric difference is why Lasso performs implicit feature selection, while Ridge usually keeps all features but reduces their influence.

Regularisation paths: what they show and why they matter

A regularisation path plots each coefficient βj\beta_jβj​ against λ\lambdaλ (often on a log scale). Interpreting these paths gives you more insight than a single best-λ\lambdaλ model:

• Ridge paths are typically smooth and continuous. As λ\lambdaλ increases, coefficients shrink steadily toward zero, often at different rates depending on feature scale and correlation structure.
• Lasso paths show coefficients that can remain at exactly zero for a long range of λ\lambdaλ, then “enter” the model at certain thresholds. As λ\lambdaλ grows, coefficients can drop to zero and effectively exit.

This path behaviour is practical: it helps you see which features are robust (survive stronger penalties) and which features only matter when the model is allowed to be more flexible.

Lasso: sparsity-inducing behaviour and its trade-offs

Lasso’s headline advantage is sparsity—it can set some coefficients to exactly zero. This is especially valuable when:

• You have many features and want a simpler model.
• You suspect only a subset of predictors truly matter.
• You want a model that is easier to explain and deploy.

However, sparsity comes with trade-offs:

1. Instability with correlated features: If two predictors carry similar information, Lasso may pick one and drop the other, and small data changes can flip that choice.
2. Bias in large effects: Lasso can shrink even strong predictors more than you might want, introducing bias to gain variance reduction.
3. Path “jumps”: Coefficients can change abruptly as λ\lambdaλ varies, which can make interpretation less smooth than Ridge.

In applied work, these trade-offs often show up when you are selecting features for forecasting or risk models—Lasso can be powerful, but you must validate stability.

Ridge: weight decay, multicollinearity support, and smoother paths

Ridge’s strength is weight decay: it spreads influence across correlated predictors rather than selecting just one. This makes Ridge attractive when:

• Predictors are strongly correlated (multicollinearity).
• You believe many features contribute a little, rather than a few contributing a lot.
• You care more about predictive performance than strict sparsity.

Ridge also tends to be numerically stable. Its smooth paths mean features rarely vanish completely; instead, their coefficients diminish as the penalty increases. In many real-world settings—marketing mix modelling, sensor data, finance-style factor modelling—this “share the load” behaviour is a major advantage taught in a data scientist course in Kolkata because it improves robustness under correlation.

Practical guidance: choosing between Lasso and Ridge

Here is a decision framework that works well in practice:

• Choose Lasso (L1) if you need a compact model, feature selection, or you have far more predictors than observations.
• Choose Ridge (L2) if multicollinearity is strong, you expect many small effects, or prediction stability matters most.
• Consider Elastic Net if you want a middle ground: it mixes L1 and L2, encouraging sparsity while handling correlated features better than pure Lasso.

For tuning, use cross-validation to choose λ\lambdaλ, and always standardise features so the penalty treats them fairly. Then inspect the regularisation path: it will tell you whether your model is relying on a few dominant predictors or many modest ones.

Conclusion

Lasso and Ridge both control overfitting, but they express different modelling beliefs. Lasso’s L1 penalty tends to create sparse solutions and clear feature selection, while Ridge’s L2 penalty provides smooth weight decay and stability—especially under correlation. If you learn to read regularisation paths, you gain a deeper diagnostic tool than “pick the best λ\lambdaλ and move on.” This skill—often emphasised in a data scientist course in Kolkata—helps you build models that are not only accurate, but also interpretable, stable, and fit for real deployment.

 

---