When Should You Use Regression Methods?

Subscribe to RichardOnData here: https://www.youtube.com/channel/UCKPyg5gsnt6h0aA8EBw3i6A?sub_confirmation=1

Topics:
0:00 - Intro
2:10 - Linear regression
7:34 - Logistic regression
8:26 - Supervised & unsupervised learning
9:09 - Inference & prediction
12:02 - Regularization

Referenced RichardOnData videos:
"Inference vs Prediction: An Overview": https://youtu.be/uh_k1jD35K8
"Interpreting Linear Model Output in R": https://youtu.be/8hmlbamLB4U
"Training and Tuning ML Models in R": https://youtu.be/rTphHY9Chgk

In this video, I discuss the practical reasons when and why you should you use regression methods (linear or logistic regression).

Let's start with the premise that we have a dataset where every cell has a value, every row is an observation, and every column is a variable. Then let's suppose we have one variable that represents some kind of "outcome" or "response" that we care about. We'll typically represent that with the letter Y. The remaining variables are interchangeably called terms like predictors, covariates, or explanatory variables, and are represented as the design matrix X or alternately as the X's.

In the simpler case where we have just two variables Y and X, we may begin to understand the relationship between these two variables by creating a scatterplot of them or looking at the correlation coefficient r. A regression model will formalize our understanding of the relationship between these variables - and the concept of "ordinary least squares" (minimizing the sum of the squared residuals, where residuals are the differences between observed and fitted values) can be projected into more than two dimensions.

More broadly speaking, regression methods are a technique typically used for "supervised learning" problems where we do have a defined response variable. Contrast this with "unsupervised learning" problems where we do not - our goal may be dimensionality reduction (e.g. principal component analysis) or defining our own classes or "clusters". In these supervised learning problems, we further have one of two goals: inference (inferring the effect a variable or variables have on the response variable), or prediction. The entire discussion above pertains to inference! It is well known that machine learning methods tend to have stronger predictive performance than statistical methods like regression.

However, regularization provides a way to balance this. There are two popular types: L2 regularization ("ridge regression") or L1 regularization ("lasso regression" or "lasso"). These models will shrink model coefficients; thereby introducing bias in exchange for reducing variance. The key difference is that the lasso can reduce coefficients to zero, thereby doubling as a feature selection approach; this does not occur with ridge regression. This can lead to a reduction in MSE; that is, predictive performance can improve - however, this is much more challenging to interpret and explain to prospective clients.

PayPal: richardondata@gmail.com
Patreon: https://www.patreon.com/richardondata
BTC: 3LM5d1vibhp1F7pcxAFX8Ys1DM6XLUoNVL
ETH: 0x3CfC599C4c1040963B644780a0E62d45999bE9D8
LTC: MH8yPjvSmKvpmRRmufofjRB9hnRAFHfx32